BitLaw

Example 48. Speech Separation

This example illustrates the application of the eligibility analysis to claims that recite artificial intelligence-based methods of analyzing speech signals and separating desired speech from extraneous or background speech. The claims, including a deep neural network, are hypothetical and are loosely based on current speech separation literature. Claim 1 is ineligible. It is "directed to" a judicial exception because it recites a judicial exception (an abstract idea), and the claim as a whole does not integrate the exception into a practical application (and thus it is directed to an abstract idea), and the claim does not amount to significantly more than the exception (does not provide an inventive concept). Claim 2 recites the same judicial exception as claim 1 but is eligible because the claim as a whole improves speech-separation technology and thus integrates the exception into a practical application of separating speech and is therefore not "directed to" the judicial exception. Claim 3 is eligible because it recites a judicial exception (an abstract idea), but the claim as a whole integrates the exception into a practical application by improving speech-to-text transcription and is therefore not "directed to" the judicial exception.

BACKGROUND

Over the last several years, there has been a vast increase in the creation and consumption of audiovisual multimedia content. Smartphones, gaming consoles, and head-mounted devices are equipped with microphones and cameras for recording. These devices also incorporate technology that allows a user to control the device with voice commands. When the same devices are used to record an event, the devices may capture voice commands as part of the recording or fail to filter the voices of nearby speakers from the recording. These recordings could be used for simple replay on social media, for personal use, or in applications like automatic captioning or transcription.

Typical human listeners can easily perceive separate sources in an acoustic mixture. For example, a typical human being is capable of paying attention to a single conversation in a crowded restaurant even though the listener is surrounded by other audible conversations. When computers interpret spoken language, they receive an audio signal from a microphone that contains all the audio the microphone has picked up. For a computer to "pay attention" to a single conversation or speaker, the relevant speech must be separated from the rest of the audio signal. Traditional computer-based speech separation techniques perform well in distinguishing and separating different classes of audio (e.g., human speech and background noise) but perform poorly in separating audio from sources belonging to the same class (e.g., speech from different speakers). Some existing solutions rely on separating speech based on volume, but a volume- based approach is unreliable in environments in which speakers may vary in how loudly they speak or in their distance from the microphone. Other solutions require training the input device to recognize a particular voice but require the user to explicitly interact with the device to provide the training data. Further, speech separation systems are ill-suited for distinguishing a conversation between individuals of interest as opposed to commands issued by a single user. As a result, using traditional speech separation techniques may cause important information that was captured during the recording to be deleted, or unwanted data not being deleted at all. When these techniques are used in pre-processing stages of speech-to-text or automatic speech recognition systems, the transcription quality suffers. There exists a need to remove these unwanted utterances from the audio, both for user privacy and for providing a high-quality recording or transcription.

Artificial neural networks (ANNs) offer a promising solution to separate speech signals from different sources. Applicant has filed a patent application that discloses a system which receives as input, a mixed speech signal x, from an audio recording device (e.g., a microphone) recording an event. The system uses a deep neural network (DNN), a type of ANN, that operates to promote separation of the features during clustering. Specifically, the DNN learns high-level feature representations of the signal x by mapping the feature representations to the embedding space. Subsequently, clustering is performed on these feature representations, with each cluster representing a distinct speech source, thereby separating speech signals of different sources s_n, where n ∈ {1, . . . N}, identified in the mixed speech signal. The signals so separated can be used for downstream applications such as transcription, removal of voice commands from a recording, and uploading into a social networking site.

The DNN could be an autoencoder, recurrent neural network, or convolutional neural network. In one embodiment, the DNN is pretrained for speech separation on audio datasets comprising speech segments from multiple speakers. The training data may comprise manually labeled audio that specifies separate speakers and separate conversations, so that the DNN can learn to distinguish both individual voices and speech context. Using these deep learning techniques has the advantage of using a simple clustering to achieve the separated speech signals of the different sources, as explained below.

Whereas a traditional Fourier transform describes the various frequency components of an entire signal, speech has frequency components that change over time. A short-time Fourier transform (STFT) is a mathematical tool for obtaining a representation of a signal when frequency components change over time by performing a sequence of Fourier transforms of smaller "windows" or "frames" of the signal. In one embodiment, a mixed speech signal x(t) is divided into T overlapping frames of equal length. This mixed speech signal is processed in the STFT domain as follows. For each frame, commonly known temporal features like pitch, variance, and zero-crossing rate are extracted and represented by a feature matrix FMtj, where t is the frame index and j is the feature index. In addition to the temporal feature extraction, a spectrogram of each frame is generated by converting the mixed audio signal into time-frequency (TF) domain S_tf by using the STFT, where f is the frequency bin index. At the end of this step, each frame is represented by X_t, which corresponds to a spectrogram Stand a corresponding row of the feature matrix FM_t.

The DNN learns the high-level feature representations of the input mixed speech signal x. Specifically, the DNN converts these feature representations X_t, obtained from the spectrograms S_t and the corresponding feature matrices FM_t, into multi-dimensional embedding vectors V. These embedding vectors V are assigned to the TF bins as a global function of the input signal (V = f_θ(X), where f_θ represents the function of the DNN). The DNN assigns embedding vectors V to each TF region such that the Euclidean distances between embedding vectors of TF bins dominated by the same source are minimized and the Euclidean distances between embedding vectors of TF bins dominated by different sources are maximized. Thus, embedding vectors V for all TF bins, representing the different sources, are calculated.

Next, clustering is performed using a k-means clustering algorithm to separate different speech sources s_n, of the mixed signal. Embedding vectors V are clustered into k distinct groups, with each group representing a distinct speech source from s_n. The clustering algorithm arbitrarily chooses k initial centers C. Then, until the algorithm converges, embedding vectors V are assigned to their closest cluster center and each center is moved to the mean of its currently assigned clustering subset. By the end of this process, the embedding vectors V are partitioned into clusters corresponding to the different constituent sources, s_n.

Binary time-frequency masks are used to separate the signals by using a binary matrix to indicate which portions of a representation should be turned on or off. In audio processing, a binary mask is a matrix of binary values that correspond to sources such that it is multiplied with a spectrogram to include or exclude portions of the audio. The binary time-frequency mask for each speaker is obtained using the cluster assignments by assigning 1 to all the TF bins corresponding to the respective speaker and 0 to the rest of the TF bins. Inverse STFT converts the obtained separated signals into the time domain. The speech signals so separated can be used for multiple different downstream applications as discussed below.

Because the DNN assigns the embedding vectors V as a function of the entire input signal, the embedding vectors V take into account the global properties of the input signal, thereby allowing the k distinct groups to correspond to the N sources identified in the mixed speech signal, thereby providing a superior speech separation. This feature of the invention is an improvement over prior speech separation methods because it allows for blind speech separation (i.e., the system is not required to have prior knowledge of the number of speakers and does not need to be trained on speech from the different constituent sources of the mixed audio signal). Due to this, the DNN could be trained with mixed speech signals comprising a fewer number of speakers and could be used to separate speech signals from a larger number of sources. Also, as this speech separation process uses both temporal and spatial features of the speech signal and derives embedding vectors V based on the global properties of the input signal, it performs well with inter-speaker variability within the same audio class for downstream applications like automatic speech recognition (ASR). Test results have consistently shown that conventional ASR systems that used the disclosed process during pre-processing stages significantly reduced the gap in transcription performance for accented speakers over traditional speech-to-text systems.

One application of this process is the removal of voice commands or background conversations from a recording of an event like a baseball game. A speech signal from an undesired source, say s_s, is identified by conventional means, for example, by obtaining an audio clip corresponding to the undesired source. In one embodiment, a user may request a separated speech signal by providing a sample of the undesired audio signal to be redacted from the mixed audio signal x. The system compares the temporal features of the sample with the source signals s_n, to find the undesired source signal ss, that is the closest match. After the inverse STFT step, the various speech signals from all the frames T are stitched together, excluding the speech signal from ss. In a preferred embodiment, an overlap-add method is used to reconstruct the entire clean speech signal. The output is therefore a clean audio signal x', such that x' includes speech signals from all the sources s_n, where n ∈ {1, . . . N}, excluding the speech signal from the source s_s. This clean audio signal x' is transmitted for storage in a remote location to be used for downstream applications like uploading into a social networking site.

Another application is real-time speech transcription or transcription of recorded audio. In one exemplary embodiment, a user may request a transcript of a desired source signal s_d, within the mixed speech signal x, during playback of a recorded audio, using a graphical user interface (GUI). After the inverse STFT step, the speech signal from only the desired source s_d is transmitted to a speech-to-text system. The ASR or speech-to-text system extracts spectral features from the desired source s_d using conventional means and generates a sequence of words, which are then converted into text. The system displays the translation result as text on the GUI.

Applicant has described that the systems discussed in the patent application could be implemented by one or more processors coupled with one or more non-transitory computer readable media. The methods described herein can be performed by execution of computer- readable instructions stored on a non-transitory computer-readable storage medium (e.g., random-access memory, flash memory, magnetic/optical storage, etc.) by the processor(s). The GUI is hardware or a combination of both hardware and software. The GUI is coupled to the systems described above and is configured to receive user instructions and outputs transcripts of audio selected by the user.

CLAIMS

[Claim 1] A speech separation method comprising:

(a) receiving a mixed speech signal x comprising speech from multiple different sources s_n, where n ∈ {1, . . . N};

(b) converting the mixed speech signal x into a spectrogram in a time-frequency domain using a short time Fourier transform and obtaining feature representation X, wherein X corresponds to the spectrogram of the mixed speech signal x and temporal features extracted from the mixed speech signal x; and

(c) using a deep neural network (DNN) to determine embedding vectors V using the formula V = f_θ(X), where f_θ(X) is a global function of the mixed speech signal x.

[Claim 2] The speech separation method of claim 1 further comprising:

(d) partitioning the embedding vectors V into clusters corresponding to the different sources s_n;

(e) applying binary masks to the clusters to create masked clusters;

(f) synthesizing speech waveforms from the masked clusters, wherein each speech waveform corresponds to a different source s_n;

(g) combining the speech waveforms to generate a mixed speech signal x' by stitching together the speech waveforms corresponding to the different sources s_n, excluding the speech waveform from a target source ss such that the mixed speech signal x' includes speech waveforms from the different sources s_n and excludes the speech waveform from the target source s_s; and

(h) transmitting the mixed speech signal x' for storage to a remote location.

[Claim 3] A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations comprising:

(a) receiving a mixed speech signal x comprising speech from multiple different sources s_n, where n ∈ {1, . . . N}, at a deep neural network (DNN) trained on source separation;

(b) using the DNN to convert a time-frequency representation of the mixed speech signal x into embeddings in a feature space as a function of the mixed speech signal x;

(c) clustering the embeddings using a k-means clustering algorithm;

(d) applying binary masks to the clusters to obtain masked clusters;

(e) converting the masked clusters into a time domain to obtain N separated speech signals corresponding to the different sources s_n; and

(f) extracting spectral features from a target source s_d of the N separated speech signals and generating a sequence of words from the spectral features to produce a transcript of the speech signal corresponding to the target source s_d.

ANALYSIS

Claim 1 is ineligible.

Claim interpretation: Under the broadest reasonable interpretation, the terms of the claim are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See Manual of Patent Examining Procedure (MPEP) 2111.

Regarding step (a), the claim does not put any limits on how the mixed speech signal is received. The broadest reasonable interpretation of a mixed speech signal encompasses audible, spoken speech from different sources. The mixed speech signal could be received with a microphone in a user device or other sensor that converts sound into an electrical signal.

Regarding step (b), the claim specifies that the mixed speech signal is converted into a spectrogram in the time-frequency domain using an STFT. A feature representation X, corresponding to the spectrogram and temporal features extracted from the mixed speech signal x, is obtained. The claim does not specify how the temporal features and the spectrogram of the mixed speech signal are obtained.

Regarding step (c), the claim specifies that a formula is used to determine embedding vectors based on the result of step (b). The claim further specifies that a DNN is used in the determination, but the claim does not include any details about the DNN or how it operates.

The broadest reasonable interpretation of claim 1 is a method of receiving spoken audio from different sources, deriving a temporal feature representation and a spectrogram of the audio, and using a DNN to calculate embedding vectors based on the temporal feature representation and a spectrogram using a mathematical formula.

Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. The claim recites the steps or acts of receiving a mixed speech signal, converting the mixed speech signal, and using a DNN to determine embedding vectors, and thus is a process (a series of steps or acts). A process is a statutory category of invention. (Step 1: YES).

Step 2A, Prong One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim "recites" a judicial exception when the judicial exception is "set forth" or "described" in the claim. The claim recites the step (b) of "converting the mixed speech signal x into a spectrogram in a time-frequency domain using an STFT, and obtaining feature representation X, wherein X corresponds to the spectrogram of the mixed speech signal x and temporal features extracted from the mixed speech signal x." In this claim, "converting" the mixed speech signal x into a spectrogram in time- frequency domain involves a mathematical operation using an STFT. Therefore, the claim recites a mathematical operation for converting a signal from one domain to another using a specific transform function. The claim also recites a step (c) that determines "embedding vectors V using the formula V = fθ(X), where fθ(X) is a global function of the input signal." The recited formula is clearly a mathematical formula or equation, and the determination is a mathematical calculation. Thus, the claim recites a mathematical formula or equation as well as a mathematical calculation, both of which fall within the mathematical concepts grouping of abstract ideas. As explained in the MPEP, when a claim recites multiple abstract ideas that fall in the same or different groupings, examiners should consider the limitations together as a single abstract idea, rather than as a plurality of separate abstract ideas to be analyzed individually. See MPEP 2106.04, subsection II.B. As the steps (b) and (c) fall within the same grouping of abstract ideas (i.e., mathematical concepts), these limitations are considered together as a single abstract idea for further analysis. (Step 2A, Prong One: YES).

Step 2A, Prong Two: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).

The claim recites a first additional element of "receiving a mixed speech signal x comprising speech from multiple different sources s_n, where n ∈ {1, . . . N}" in step (a). As explained above, step(a) is claimed at a high level of generality and could describe receiving the mixed speech signal with a microphone in a user device or other sound sensor. The element amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception to perform the calculation (i.e., converting the mixed speech signal into a time-frequency domain representation). The "receiving" element does not impose any other meaningful limits on the claim. Therefore, the additional limitation is insignificant extra-solution activity. See MPEP 2106.05(g).

The method also recites a second additional element in step (c) of "using a deep neural network (DNN) to determine embedding vectors V . . ." When determining whether a claim simply recites a judicial exception with the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners may consider: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. See MPEP 2106.05(f). Here, there are no details about a particular DNN or how the DNN operates to derive the embedding vectors other than that it is being used to determine the embedding vectors. The DNN is used to generally apply the abstract idea (i.e., perform the mathematical calculation using the recited mathematical equation) without placing any limitation on how the DNN operates to derive the embedding vectors as a function of the input signal. In addition, the limitation recites only the idea of determining embedding vectors using a DNN without details on how this is accomplished. The claim omits any details as to how the DNN solves a technical problem, and instead recites only the idea of a solution or outcome. Also, the claim invokes a generic DNN merely as a tool for making the recited mathematical calculation rather than purporting to improve the technology or a computer. See MPEP 2106.05(f). Therefore, the limitation represents no more than mere instructions to apply the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers.

The disclosure identifies a technical problem encountered in the field of speech-separation and provides this invention as a solution to the identified speech-separation problem. The disclosure clearly describes how this invention offers an improvement over existing speech-separation methods by providing a particular speech-separation technique that solves the problem of separating speech from different speech sources belonging to the same class, while not requiring prior knowledge of the number of speakers or speaker-specific training. In particular, the improvement is achieved by determining embedding vectors as a function of the input signal, partitioning those vectors into clusters, and synthesizing a reconstructed mixed speech signal based on these clusters. The claim, however, only requires determining the embedding vectors and therefore does not reflect the improvement discussed in the disclosure. The recited generic DNN merely adds a generic computer component to perform the method and therefore fails to provide an improvement to the technology or technical field. See MPEP 2106.05(a). Even when viewed in combination, these additional elements do not integrate the recited judicial exception into a practical application (Step 2A, Prong Two: NO), and the claim is directed to the judicial exception. (Step 2A: YES).

Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05.

At Step 2A, Prong Two, the second additional element in step (c), "using a deep neural network," was found to represent no more than mere instructions to apply the judicial exception on a computer using generic computer components. The analysis under Step 2A, Prong Two is carried through to Step 2B. Further, the first additional element in step (a) was found to be insignificant extra-solution activity. However, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B. See MPEP 2106.05, subsection I.A. At Step 2B, the re-evaluation of the insignificant extra-solution activity consideration takes into account whether or not the extra-solution activity is well understood, routine, and conventional in the field. See MPEP 2106.05(g). Here, the step of receiving a mixed speech signal is mere data gathering that is recited at a high level of generality, and as discussed in the disclosure, is well-understood (e.g., the first paragraph of the background explains that smartphones and other devices have long been equipped to receive a mixed speech signal via the microphones integrated into the devices). Therefore, this limitation remains insignificant extra-solution activity even upon reconsideration and does not amount to significantly more.

Even when considered in combination, these additional elements represent mere instructions to apply an exception and insignificant extra-solution activity, and therefore do not provide an inventive concept (Step 2B: NO). The claim is not eligible.

Claim 2 is eligible.

Claim interpretation: Under the broadest reasonable interpretation, the terms of the claim are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP 2111. Claim 2 is a dependent claim that depends from and requires all the limitations of claim 1.

Regarding step (d), the claim does not place any limits on how the embedding vectors are partitioned into clusters corresponding to the different sources. The clustering could be performed using a k-means algorithm as indicated in the disclosure or any other algorithm known to a person of ordinary skill in the art.

Step (e) requires applying binary masks to the clusters. The plain meaning of "applying a binary mask" to a person of ordinary skill in the art is a mathematical operation of using a binary matrix to indicate which portions of a representation should be turned on or off. Such masking can be performed in any way known in the art, for example, by performing bitwise operations on two numbers or multiplying a binary matrix with another numerical representation, etc.

Step (f) synthesizes the speech waveforms from the masked clusters, where each waveform corresponds to a different source of the mixed speech signal. The background section explains that this synthesis is performed by converting the masked clusters into separate speech signals in the time domain, corresponding to the different sources in the mixed speech signal, using inverse STFT.

Step (g) combines the separated speech waveforms of step (f) by stitching together the speech waveforms corresponding to the different sources s_n, excluding the speech waveform from the target source s_s such that the resultant mixed speech signal excludes at least one speech signal from a target source and includes speech signals from other sources. The background section indicates that an overlap-add method is used to perform the stitching to reconstruct the entire clean speech signal, but step (g) is broad enough to cover any method known in the art to perform the stitching.

Step (h) specifies that the reconstructed mixed speech signal is transmitted for storage to a remote location. Note that this step does not require that the reconstructed mixed speech signal is actually stored, but that the signal is transmitted with the intended result of being stored at a remote location.

As described above with respect to claim 1, the broadest reasonable interpretation of claim 2 is a method of receiving spoken audio from different sources, deriving a temporal feature representation and a spectrogram of the audio, and using a DNN to calculate embedding vectors based on the temporal feature representation and a spectrogram using a mathematical formula. The embedding vectors are then partitioned into clusters, the clusters are modified using binary masks, and the modified clusters are synthesized into separate speech signals. A new, combined mixed speech signal is created by excluding at least one speech signal from one source and including speech signals from other sources. The combined mixed speech signal is then transmitted.

Step 1: As discussed above with respect to claim 1, the claim recites the steps or acts of receiving a mixed speech signal, converting the mixed speech signal, and using a DNN to determine embedding vectors. Dependent claim 2 adds further steps of partitioning the embedding vectors into clusters, applying binary masks to the clusters and synthesizing the result, combining the resultant signals into a mixed speech signal, and transmitting the mixed speech signal. Claim 2 is therefore a process (a series of steps or acts). A process is a statutory category of invention. (Step 1: YES).

Step 2A, Prong One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. MPEP 2106.04, subsection II explains that a claim "recites" a judicial exception when the judicial exception is "set forth" or "described" in the claim. As discussed above with respect to claim 1, steps (b) and (c) recite mathematical concepts. Step (d) recites "partitioning the embedding vectors V into clusters corresponding to the different sources s_n." The claim places no limits on how this partitioning is performed. That is, nothing in the claim element precludes the step from practically being performed in the mind. For example, "partitioning . . . into clusters" encompasses a human arbitrarily selecting groups of vectors and mentally assigning them to clusters. The recitation of a DNN in this claim does not negate the mental nature of these limitations because the claim here merely uses the DNN as a tool to perform the otherwise mental process. See MPEP 2106.04(a)(2), subsection III.C. The claim therefore recites a mental process.

The claim also recites step (e) - "applying binary masks to the clusters to create masked clusters." This step recites a mathematical operation for generating numbers based on binary calculations. Therefore, the claim recites a further mathematical calculation, which falls within the mathematical concept grouping of abstract ideas.

"Unless it is clear that a claim recites distinct exceptions, such as a law of nature and an abstract idea, care should be taken not to parse the claim into multiple exceptions, particularly in claims involving abstract ideas." MPEP 2106.04, subsection II.B (discussing Bilski v. Kappos, 561 U.S. 593 (2010)). Thus, if possible, the examiner should consider the limitations together as a single abstract idea rather than as a plurality of separate abstract ideas to be analyzed individually. As discussed above, steps (b), (c), and (e) of claim 2 recite mathematical concepts and step (d) recites a mental process. An examiner should identify the claim as reciting both a mental process and a mathematical concept for Step 2A, Prong One, and consider the limitations (b)-(e) together as a single abstract idea for further analysis. See MPEP 2106.04, subsection II.B.

Step (f) synthesizes the speech waveforms from the masked clusters, where each waveform corresponds to a different source of the mixed speech signal. This step requires converting the result of step (e) into separate speech signals in the time domain. Synthesizing speech waveforms from a cluster of numbers is not a process that can be practically performed in the human mind. Further, while the synthesis would involve a mathematical calculation, the claim does not specify any mathematical formula, calculations, or relationships. In addition, step (f) is not a method of organizing human activity because it does not fall within the enumerated sub-groupings of fundamental economic principles or practices, commercial or legal interactions, and managing personal behavior and relationships or interactions between people.

Similarly, step (g) combines the speech waveforms to generate a mixed speech signal by stitching together speech waveforms corresponding to the different sources excluding the speech signal from a target source. Even though the disclosure explains that the stitching could be performed by an overlap-add method, which is a mathematical operation, the claim recites no details of how the stitching is performed. Additionally, while the claim recites variables, variables on their own are not mathematical relationships, formulas, or calculations. Therefore, the combining step is merely based on or involves a mathematical concept but does not recite a mathematical concept. See MPEP 2106.04(a)(2), subsection I. Generating a mixed speech signal such that it includes speech signals from different sources and excludes the speech signal from a target source is not a process that can be practically performed in a human mind. Therefore, step (g) is neither a mathematical concept nor a mental process. In addition, step (g) is not a method of organizing human activity because it does not fall within the enumerated sub-groupings.

As discussed above, limitations (b)-(e) are considered as a single abstract idea, and the claim is considered as reciting limitations which fall within the mathematical concepts grouping of abstract ideas. (Step 2A, Prong One: YES).

Step 2A, Prong Two: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).

As discussed above with respect to claim 1, step (a) recites a data gathering step of receiving a mixed speech signal, and step (c) recites using a DNN to determine embedding vectors, which is equivalent to the words "apply it."

Step (h) recites "transmitting the mixed speech signal x' for storage to a remote location." As discussed above, the mixed speech signal x' is transmitted with the intended result of being stored at a remote location for future use, for example, for audio playback or to be uploaded into a social media website. This limitation is merely a post-solution step of transmitting data output—a nominal addition to the claim that does not meaningfully limit the claim. Therefore, step (h) is an insignificant extra-solution activity. See MPEP 2106.05(g). The remaining additional limitations beyond the abstract idea recited in limitations (b), (c), (d), and (e) are synthesizing speech waveforms from the masked clusters, recited in limitation (f), and generating the mixed speech signal excluding the speech signal from a target source, recited in limitation (g).

Step (f) recites "synthesizing speech waveforms from the masked clusters, wherein each speech waveform corresponds to a different source s_n," and step (g) recites "combining the speech waveforms to generate a mixed speech signal x' by stitching together the speech waveforms corresponding to the different sources s_n, excluding the speech waveform from a target source s_s such that the mixed speech signal x' includes speech signals from the different sources s_n, where n ∈ {1, . . . N}, and excludes the speech signal from the target source ss." Steps (f) and (g) integrate the abstract idea into a practical application. The disclosure explains that devices that capture audio cannot properly distinguish different speech sources belonging to the same class and that the current available solutions do not adequately address this problem because they require a target user, whose speech is to be recognized, to explicitly interact with the device to provide training data. The disclosure states that this invention offers an improvement over existing speech-separation methods by providing a particular speech-separation technique that solves the problem of separating speech from different speech sources belonging to the same class, while not requiring prior knowledge of the number of speakers or speaker-specific training. The claim reflects the improvement discussed in the disclosure by reciting details of how the DNN aids in the cluster assignments to correspond to the sources identified in the mixed speech signal, which are then synthesized into separate speech waveforms in the time domain and converted into a mixed speech signal, excluding audio from the undesired source. See MPEP 2106.05(a). While on their own steps (b)-(e) recite judicial exceptions, steps (f) and (g) are directed to creating a new speech signal that no longer contains extraneous speech signals from unwanted sources. The claimed invention reflects this technical improvement by including these features. Further, converting clusters into separate speech waveforms and generating a mixed speech signal from the separate speech waveforms are not insignificant extra-solution activity, mere instructions to apply the exception, or mere field of use limitations. Rather, these steps reflect the improvement described in the disclosure. Accordingly, the claim is directed to an improvement to existing computer technology or to the technology of speech separation, and the claim integrates the abstract idea into a practical application. (Step 2A, Prong Two: YES). The claim is eligible. (Step 2A: NO).

Claim 3 is eligible.

Claim interpretation: Under the broadest reasonable interpretation, the terms of the claim are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP 2111. The preamble specifies that the claim is to a non-transitory computer-readable storage medium containing instructions that when executed by the one or more processors with which it is associated, cause the processor to perform the receiving, generating, and producing steps recited in the claim. The disclosure gives random- access memory, flash memory, magnetic/optical storage, etc. as examples of a non-transitory computer-readable storage medium but the claim does not specify the type of non-transitory computer-readable storage medium.

Regarding step (a), the claim does not place any limits on how the mixed speech signal is received. The broadest reasonable interpretation of a mixed speech signal encompasses audible, spoken speech from different sources. The mixed speech signal could be received with a microphone in a user device or other sensor that converts sound into an electrical signal. The claim requires a DNN that receives this mixed speech signal to be trained for source separation.

Regarding step (b), the claim specifies that the DNN is used to convert a time-frequency representation of the mixed speech signal to embeddings in the feature space but does not provide any structural details about the DNN itself. The specification discusses processing the mixed speech signal in STFT domain to obtain temporal features and spectrograms, which are then used by the DNN to determine embeddings V in the feature space as a function of the input signal. As the claim does not specify how the time-frequency representation of the mixed speech signal is obtained or how it is converted into embeddings as a function of the signal x, this sub- step can be performed as indicated in the specification or by any other method known to a person having ordinary skill in the art.

Step (c) requires clustering the embeddings using a k-means clustering algorithm but does not place any limits on how the algorithm is implemented. The clustering could be performed using the k-means algorithm either as described in the disclosure or in other ways known to a person having ordinary skill in the art.

Step (d) obtains masked clusters by applying binary masks to the clusters. The plain meaning of "applying a binary mask" to a person of ordinary skill in the art is a mathematical operation of using a binary matrix to indicate which portions of a representation should be turned on or off. Such masking can be performed in any way known in the art, for example, by performing bitwise operations on two numbers or multiplying a binary matrix with another numerical representation, etc.

Step (e) requires converting the masked clusters into separate speech signals in the time domain, corresponding to the different sources in the mixed speech signal. The claim does not specify how the conversion is performed.

The final step (f) requires extracting spectral features from only one target source s_d of the N separated signals from the output of step (e) and generating a sequence of words from the spectral features to produce a transcript of the speech signal corresponding to the target source sd. The disclosure states that the desired speech signal from the output of the inverse STFT step is transmitted to an ASR, which extracts spectral features from the desired source s_d using conventional means and generates a sequence of words, which are then converted into text. The claim does not specify a particular way in which the extracting and generating is done; therefore, the step of extracting the spectral features and generating a sequence of words can be implemented by ASR systems known in the art to produce the transcript.

Therefore, the broadest reasonable interpretation of claim 3 is a non-transitory computer- readable storage medium that stores instructions, which when executed by a processor, cause the processor to perform the steps of receiving a mixed speech signal constituting audio from different sources by a DNN that calculates embedding vectors from time-frequency representation of the signal. The embeddings are then partitioned into clusters and the clusters are converted into separate speech signals in the time domain. Only one specific separated speech signal of these separated speech signals is converted into text to produce a transcript.

Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. The preamble specifies that the claim is to a non-transitory computer-readable medium, which causes the one or more processors it is associated with to perform a series of steps. The disclosure gives random-access memory, flash memory, and magnetic/optical storage as non-limiting examples of a non-transitory computer-readable storage medium. The broadest reasonable interpretation of the claim covers only statutory embodiments of a computer-readable medium in light of the disclosure and not a transitory signal. A non- transitory computer-readable storage medium falls within the "manufacture" category of invention. (Step 1: YES).

Step 2A, Prong One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim "recites" a judicial exception when the judicial exception is "set forth" or "described" in the claim. Step (b) requires converting a time-frequency representation of the mixed speech signal into embeddings in a feature space as a function of the mixed speech signal, which is a mathematical equation written in text format. Step (c) requires clustering the embeddings by a k-means clustering algorithm, which is a mathematical calculation. Step (d) obtains masked clusters by applying binary masks to the clusters, which is also a mathematical calculation. Thus, the claim recites mathematical calculations which fall within the mathematical concepts grouping of abstract ideas.

Step (e) requires converting a cluster of points in feature space into speech signals in the time domain, which is not a process that can be practically performed in the human mind. Further, while the conversion may be based on mathematical concepts, the claim does not specify any mathematical formulae, calculations, or relationships.

Finally, step (f) requires extracting spectral features from only one target source s_d of the N separated signals from the output of step (e) and generating a sequence of words from the spectral features to produce a transcript of the speech signal corresponding to the target source sd. Extracting spectral features from a signal and generating a sequence of words from these extracted features to produce a transcript is not a process that can practically be performed in the human mind. While this extraction and generation involves mathematical operations, the claim does not specify any mathematical formulae, calculations, or relationships. Neither do steps (e) and (f) fall within the enumerated sub-groupings of a method of organizing human activity. Therefore, steps (e) and (f) do not recite a judicial exception.

As explained in the MPEP, when a claim recites multiple abstract ideas that fall in the same or different groupings, examiners should consider the limitations together as a single abstract idea rather than as a plurality of separate abstract ideas to be analyzed individually. See MPEP 2106.04, subsection II.B. As discussed above, limitations (b)-(d) recite mathematical concepts. As all these steps (b)-(d) fall within the same grouping of abstract ideas (i.e., mathematical concepts), these limitations are considered together as a single abstract idea for further analysis. (Step 2A, Prong One: YES).

Step 2A, Prong Two: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).

Claim 3 recites as step (a) the additional limitation of "receiving a mixed speech signal x comprising speech from multiple different sources sn , where n ∈ {1, . . . N}." As explained in the claim interpretation portion above, the limitation is claimed at a high level of generality and could describe receiving the mixed speech signal with a microphone in a user device or other sound sensor. The limitation amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception to perform the calculations of steps (b), (c), and (d). The limitation does not impose any other meaningful limits on the claim. Therefore, this additional limitation is insignificant extra-solution activity. See MPEP 2106.05(g).

Another additional limitation beyond the abstract idea recited in step (b) is the use of a DNN trained on source separation. When determining whether a claim simply recites a judicial exception with the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, examiners may consider: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. See MPEP 2106.05(f). Here, the claim recites no details about a particular DNN. The DNN is used to generally apply the abstract idea (i.e., perform the mathematical calculation recited in step (b)) without placing any limitation on how the DNN operates to derive the embedding vectors. In addition, the limitation would cover every mode of implementing the recited abstract idea using a DNN. The claim omits any details as to how the DNN solves a technical problem and instead recites only the idea of a solution or outcome. See MPEP 2106.05(f). Therefore, the limitation represents no more than mere instructions to implement the abstract idea recited in step (b), which is equivalent to adding the words "apply it" to the recited judicial exception. In addition, the claim confines the use of the judicial exception recited in step (b) to the technological environment of a DNN by generally linking the use of the judicial exception to the recited DNN. Therefore, this general DNN recitation does not integrate the judicial exception into a practical application. See MPEP 2106.05(h). Therefore, it can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to a particular field of use or a technological environment.

The remaining additional limitations are step (e), which converts the masked clusters into N separate speech signals in time domain, and step (f), which extracts spectral features from only one target source s_d of the N separate signals from the output of step (e) and generates a sequence of words from the spectral features to produce a transcript. These additional limitations integrate the abstract idea recited in steps (b), (c), and (d) into a practical application of speech-to-text conversion.

The disclosure explains that devices that capture audio perform poorly in distinguishing conversations between individuals of interest from unwanted utterances due to their inability in distinguishing different speech sources belonging to the same class, thereby resulting in poor quality transcriptions of the recorded speech. The disclosure states that this invention offers an improvement over existing speech-separation methods by providing a particular speech- separation technique that solves the problem of separating speech from different speech sources belonging to the same class, and also performing well with inter-speaker variability within the same audio class for transcriptions. The disclosure states that the invention derives embeddings by the DNN based on the global properties of the input signal, which is an improvement over prior art speech separation methods. In addition, the invention uses both temporal and spatial features of the speech signal; this feature of the invention helps a downstream conventional speech-to-text system reduce the gap in transcription performance for accented speakers over traditional speech-to-text methods.

Here, the claim reflects these technical improvements discussed in the disclosure by reciting details of how the DNN trained on source separation aids in the cluster assignments to correspond to the sources identified in the mixed speech signal, which are then converted into separate speech signals in the time domain to generate a sequence of words from the spectral features, thereby making individual transcription of each separated speech signal possible. See MPEP 2106.05(a).

While on their own, the steps (b), (c), and (d) recite an abstract idea, the ordered combination of the steps of receiving a mixed speech signal, processing the speech signal to produce masked clusters, converting the masked clusters into separate signals in time domain, extracting spectral features from one such converted signal, and generating a sequence of words from the extracted spectral features to produce a transcript reflects the technical improvement discussed in the disclosure. Accordingly, the claim is directed to an improvement to existing speech-to-text technology, and the claim integrates the abstract idea recited in steps (b), (c), and (d) into a practical application of speech-to-text conversion of a speech signal corresponding to one source of the mixed speech signal. Thus, the claim as a whole integrates the judicial exception into a practical application (Step 2A, Prong Two: YES), such that the claim is not directed to the judicial exception. (Step 2A: NO). The claim is eligible.