A Speech-to-Text Translation System for English–Malay with Informal and Code-Switched Speech

1.1Background

Automatic speech recognition (ASR) and machine translation (MT) have matured considerably over the past decade, driven by deep learning advances and the availability of large-scale annotated datasets. Modern ASR systems such as OpenAI's Whisper [12] and Meta's wav2vec 2.0 [13] achieve near-human performance on benchmark datasets such as LibriSpeech. Similarly, neural machine translation (NMT) systems built on transformer architectures have largely superseded earlier phrase-based statistical approaches, with state-of-the-art systems such as NLLB-200 [14] supporting hundreds of language pairs. These technologies underpin widely used applications, voice assistants, automatic subtitling, and real-time translation services that are increasingly embedded in everyday life.

Despite this progress, a persistent and underexplored gap exists between laboratory-level performance and real-world usability. Most benchmark datasets consist of clean, formal, monolingual speech read by native speakers in controlled recording environments. Conversational speech, by contrast, is characterised by disfluencies, informal vocabulary, reduced articulation, and, in multilingual communities, frequent switching between languages. Current ASR and MT models trained primarily on formal monolingual data degrade significantly under these conditions [1], [6].

Nowhere is this gap more evident than in Malaysia. Informal spoken communication in Malaysia frequently involves Manglish, a colloquial variety of Malaysian English characterised by Malay and Chinese lexical borrowings, local discourse particles such as lah, ah, wah, and kan, non-standard grammar, and extensive code-switching between English and Malay [9]. While Manglish is the dominant register of everyday spoken interaction among educated Malaysians, it is not well represented in any publicly available speech recognition or translation dataset, and most commercial speech systems are poorly equipped to handle it.

1.2Problem Statement

The core problem addressed by this project is the poor performance of existing, off-the-shelf speech-to-text translation systems when applied to informal, code-switched Malaysian English speech. Specifically, two challenges are intertwined:

1. 1.ASR accuracy on Manglish and code-switched speech: Models pretrained on formal English corpora cannot reliably transcribe Malay words, Manglish particles, or code-switched utterances, resulting in high Word Error Rates [1], [6], [7], [17].
2. 2.Cascading errors into MT: Errors at the ASR stage are passed as input to the MT component. Since MT models are trained on clean, grammatical text, distorted ASR outputs further degrade translation quality, a phenomenon known as error propagation or cascading error amplification [3], [18].

Understanding the magnitude and nature of these challenges is a prerequisite for building better multilingual speech translation systems suited to the Malaysian context.

1.3Project Approach

Rather than developing new models from scratch, this project adopts a system-oriented evaluation approach. A cascaded pipeline consisting of an ASR component followed by an MT component is constructed using pretrained models, and performance is rigorously evaluated on a purpose-built dataset of informal Malaysian speech. A controlled fine-tuning experiment on wav2vec 2.0 is conducted to investigate whether limited domain adaptation can meaningfully improve ASR performance under low-resource constraints.

This approach is grounded in the observation, supported by prior literature [1], [6], that systematic evaluation of existing models under realistic conditions is itself a valuable contribution, particularly where benchmark datasets do not reflect real-world usage. Rather than optimising for a single model, the project generates comparative evidence about the strengths and limitations of different ASR configurations and their impact on end-to-end translation quality.

1.4Research Objectives

The main objectives of this project are as follows:

1. 1.To design and implement a speech-to-text translation pipeline that converts spoken English (including Manglish) into written Malay.
2. 2.To evaluate the performance of existing ASR and MT models when applied to informal Malaysian speech, particularly Manglish and code-switched utterances.
3. 3.To investigate the impact of limited domain adaptation through fine-tuning, by comparing pretrained ASR models against lightly fine-tuned variants.
4. 4.To analyse recognition and translation errors arising from informal language use, dialectal variation, and code-switching, using both quantitative metrics and qualitative analysis.
5. 5.To assess the cascading effect of ASR errors on downstream machine translation quality.

1.5Scope

The project is scoped to focus on feasibility and analytical depth within available time and resource constraints. It covers:

● Evaluation of four ASR configurations (Whisper, wav2vec baseline, fine-tuned wav2vec, wav2vec + LM) ● End-to-end translation quality evaluation using a shared MT component (NLLB-200) ● A manually constructed dataset of 61 informal English–Malay code-switched utterances ● Evaluation using WER and BLEU metrics, supplemented by qualitative error analysis

The scope explicitly excludes: model training from scratch, real-time or streaming speech processing, speech synthesis (text-to-speech), and large-scale deployment.

1.6Dissertation Structure

The remainder of this dissertation is organised as follows. Chapter 2 reviews related literature on ASR architectures, code-switching challenges, and English–Malay machine translation. Chapter 3 describes the system architecture, dataset, models, and evaluation methodology. Chapter 4 presents and interprets all results, including quantitative metrics and qualitative error analysis. Chapter 5 concludes the dissertation with a summary of contributions, limitations, and directions for future work.

2.1Automatic Speech Recognition: Evolution and Modern Architectures

The landscape of ASR has been transformed by deep learning approaches. Early systems relied on Gaussian mixture model–hidden Markov model (GMM-HMM) pipelines that required carefully engineered acoustic and language models. The introduction of deep neural networks, and subsequently end-to-end models based on connectionist temporal classification (CTC) and attention mechanisms, substantially simplified the training pipeline while improving accuracy [6].

2.1.1wav2vec 2.0

wav2vec 2.0 [13] is a self-supervised speech representation model developed by Meta AI. The model architecture consists of a convolutional feature encoder that maps raw audio waveforms

into a sequence of latent representations, followed by a transformer-based context network. During pretraining, the model is trained to solve a contrastive task over quantised speech representations, using large quantities of unlabelled audio data. This pretraining strategy allows the model to learn rich acoustic representations without requiring transcribed speech, making it particularly valuable in low-resource settings.

After pretraining, wav2vec 2.0 can be fine-tuned for speech recognition on labelled data using CTC loss. The publicly available facebook/wav2vec2-base-960h model used in this project is pretrained on LibriSpeech 960 hours of clean English audiobook speech [13] and achieves strong performance on standard English benchmarks. However, because its training corpus consists entirely of formal, clean, monolingual English, its generalisation to informal conversational speech and code-switched input is limited [8]. This highlights a key limitation when applying such models to real-world multilingual contexts such as Manglish.

Research has shown that targeted fine-tuning of wav2vec 2.0 on domain-specific data can yield meaningful improvements in low-resource settings [11]. Anidjar et al. [11] demonstrate that augmenting wav2vec 2.0 with task-specific data and shallow language model integration improves recognition of under-resourced languages, motivating the fine-tuning experiment conducted in this project.

2.1.2OpenAI Whisper

Whisper [12] takes a fundamentally different approach to ASR. Rather than self-supervised pretraining, Whisper is trained end-to-end using a large supervised dataset of 680,000 hours of weakly labelled multilingual audio collected from the internet. The model uses a sequence-to-sequence transformer architecture (encoder–decoder), with the encoder processing log-Mel spectrogram features and the decoder generating text autoregressively.

A key strength of Whisper is its multilingual capability. The training data covers 99 languages, including Malay, and the model has been shown to be robust to accents, background noise, and informal speech. Several model sizes are available; this project uses the small variant (39 million parameters), which balances accuracy and computational efficiency. Despite its broad training, prior work has shown that Whisper's performance degrades notably on code-switched speech [7], and that it sometimes normalises or translates Malay words into English equivalents rather than transcribing them faithfully, a behaviour with important consequences for downstream Malay translation. However, Whisper is not explicitly optimised for code-switched speech, and this tendency to alter non-English segments can distort the original linguistic structure, negatively affecting downstream tasks such as machine translation where fidelity to the source text is critical.

Yang et al. [7] demonstrate that Whisper experiences significant degradation on Mandarin–English code-switched speech, while Agro et al. [6] report similar performance drops across multiple language pairs. However, these studies primarily evaluate relatively structured forms of code-switching. In contrast, Malay–English code-switching involves informal discourse particles and non-standard grammar [9], suggesting that degradation in Manglish may be more severe than prior work indicates.

A similar pattern is therefore anticipated for Malay–English code-switching, although the presence of discourse particles and informal grammar introduces additional challenges not present in the Mandarin–English setting [9].

2.2Code-Switching in Natural Language Processing

Code-switching refers to the alternation between two or more languages within a single conversation or utterance. It is a natural communicative behaviour in multilingual communities and reflects a speaker's linguistic competence rather than a deficit [9]. Winata et al. [1] provide a comprehensive survey of code-switching research in NLP, noting that while substantial progress has been made in text-based code-switching tasks, speech-based code-switching and low-resource language pairs remain significantly under-served.

For ASR systems, code-switching poses a particular challenge. Models typically assume a fixed language throughout an utterance and rely on language-specific acoustic and language models. When the speaker switches languages, the model encounters phonetic patterns and vocabulary outside its training distribution, leading to increased substitution and deletion errors [6]. The abrupt nature of language switching, sometimes occurring mid-word or mid-phrase, further complicates model adaptation [2].

Agro et al. [6] conduct a systematic literature review of code-switching in end-to-end ASR, confirming that even large pretrained models experience notable WER degradation on code-switched input. The review highlights the lack of standardised code-switched evaluation datasets as a key obstacle to progress, which directly motivates the manual dataset construction strategy adopted in this project. However, most existing studies focus on relatively structured or high-resource code-switching scenarios, whereas informal variants such as Manglish introduce additional complexity due to non-standard grammar, pronunciation variation, and discourse particles.

These findings indicate that code-switching introduces both acoustic and linguistic variability that is not adequately captured by models trained on monolingual data, highlighting a key gap addressed in this work.

2.2.1Malay–English Code-Switching

Treffers-Daller et al. [9] analyse the specific linguistic patterns of Malay–English code-switching, noting that English serves as the matrix language in most informal interactions among educated Malaysians, with Malay lexical insertions and function words embedded within English syntactic frames. Discourse particles such as lah, ah, kan, wah, and mah are particularly common and carry pragmatic meanings such as assertion softening, seeking agreement, and expressing resignation. These particles have no direct equivalents in English and are typically absent from both ASR training data and MT training corpora, making them sources of consistent error at both pipeline stages.

The informal variety of Malaysian English also exhibits non-standard grammar patterns, including topic-prominent constructions ("this one can"), copula dropping ("he very smart"), and imperative softening through discourse markers. These features place Manglish at considerable distance from the standard English that most ASR and MT models assume.

2.3Low-Resource and Dialect Speech Recognition

For low-resource languages such as Malay, the scarcity of annotated speech data significantly constrains the development of high-quality ASR systems. Tan et al. [4] investigate low-resource Malay dialect speech recognition using transfer learning from standard Malay models, finding that recognition error rates increase substantially for dialectal speech even when acoustic conditions are controlled. This observation directly parallels the challenges of Manglish recognition, where informal vocabulary and phonological reduction introduce recognition difficulty similar to dialectal variation.

Adila et al. [8] examine the performance of multilingual models including Whisper on Indonesian speech, a language closely related to Malay, under informal and spontaneous conditions. They find that model performance degrades substantially under these conditions and that fine-tuning on domain-specific data yields measurable improvements, supporting the fine-tuning strategy adopted in this project. However, their study focuses on Indonesian rather than Malaysian English and does not account for discourse particles or code-switched structures that are characteristic of Manglish.

Bansal et al. [3] discuss the broader challenges of low-resource speech-to-text translation, noting that limited data availability forces a trade-off between model complexity and overfitting risk. Their work motivates the conservative fine-tuning design used here — specifically, the decision to fine-tune only the uppermost encoder layers of wav2vec 2.0 rather than the full model, and to use a small number of epochs with a low learning rate. Their work assumes monolingual input and does not consider code-switched speech, limiting its applicability to

scenarios where ASR input itself is bilingual.

2.4Language Model Integration in ASR

A traditional approach to improving CTC-based ASR is shallow fusion with an external n-gram language model (LM) during beam search decoding. The LM provides prior probabilities over word sequences, biasing the decoder towards more linguistically probable outputs. KenLM [15] is a widely used efficient n-gram language modelling toolkit that can be integrated with CTC decoders through libraries such as pyctcdecode.

However, LM integration introduces its own challenges. In the present project's setting, the vocabulary mismatch between a standard English n-gram LM and the Manglish vocabulary of the test utterances means that the LM may penalise code-switched words as unlikely, worsening performance. Furthermore, incorrect LM beam search configurations can lead to pathological outputs such as word-merging (where word boundaries are lost), as observed in this project's results.

Nguyen and Tran [2] propose asynchronous text–speech adaptation approaches for code-switched ASR and find that shallow LM fusion without targeted adaptation of the LM vocabulary to the code-switched domain frequently degrades performance on mixed-language input, consistent with the findings reported here. This prior evidence motivated the decision to treat KenLM integration as a hypothesis to test rather than a guaranteed improvement, and informed the expectation that vocabulary mismatch would be a primary failure mode.

2.5English–Malay Machine Translation

Neural machine translation for English–Malay has been studied to a limited extent compared to higher-resource language pairs. Phua et al. [5] investigate sequence-to-sequence NMT for English–Malay, finding that attention-based transformer models substantially outperform statistical MT approaches, with performance typically evaluated via BLEU scores [16].

However, their evaluation assumes clean, grammatical English input and does not consider the effect of noisy, code-switched ASR output on translation quality.

The NLLB-200 model [14] (No Language Left Behind), developed by Meta AI, represents the current state of the art for multilingual translation. The facebook/nllb-200-distilled-600M variant used in this project is a distilled model supporting 200 language pairs, including English–Malay (identified as zsm_Latn in the model's language taxonomy). Despite its broad multilingual coverage, the model was trained on formal, clean parallel text data, meaning it may struggle with informal Manglish input even when ASR transcription is reasonably accurate.

A critical consideration for this project is the behaviour of MT models when presented with noisy ASR output. When ASR errors distort words to the point of unrecognisability such as "susah sikit" (a little difficult) being transcribed as "susasi kid" — the MT model cannot recover the original meaning and produces nonsensical translations. This error propagation effect has been highlighted as a fundamental limitation of cascaded speech translation architectures [5], [6].

2.6Summary and Research Gap

Prior work on code-switched ASR [1], [6], [7] has focused predominantly on structured bilingual datasets in high-resource settings. Studies on Malay speech [4], [8] primarily address dialectal or Indonesian variants rather than informal Malaysian English. Similarly, English–Malay machine translation [5] has been evaluated on clean, standard text, without considering noisy, code-switched ASR input.

While the cascading error problem in ASR–MT pipelines has been acknowledged in prior work [5], [6], it has not been empirically examined in the context of informal Manglish speech. In particular, the interaction between ASR errors and MT performance in a realistic, code-switched Malaysian setting remains underexplored.

This project addresses that specific gap by evaluating a cascaded ASR–MT pipeline on informal English–Malay speech and analysing how recognition errors propagate into downstream translation under real-world conditions.

3.1System Architecture

The system implements a cascaded speech-to-text translation pipeline consisting of two sequentially connected modules:

1. 1.ASR Module: Converts raw audio input to text transcription in the source language (English or code-switched English–Malay).
2. 2.MT Module: Translates the ASR-produced text from English (or Manglish) into Malay.

This two-stage architecture allows the contribution of each component to overall system performance to be measured independently. The ASR output can be evaluated with WER against reference transcriptions, and the translation output can be evaluated with BLEU against reference Malay translations. The gap between the two stages exposes the cascading error effect quantitatively.

The pipeline was intentionally kept modular. The MT component is held fixed across all experimental conditions, so any differences in end-to-end BLEU scores between pipeline variants can be attributed solely to differences in ASR output quality.

Figure 3.1:1 illustrates the overall architecture of the proposed system.

3.2Dataset

3.2.1Dataset Construction

Given the limited availability of publicly accessible code-switched speech datasets for Malaysian English, a bespoke dataset was manually constructed for this project. The dataset consists of 61 utterances designed to reflect informal, conversational Malaysian English speech as used in everyday settings such as university campus communication, group study sessions, and casual discussion.

A total of 21 utterances were recorded by the author, with the remaining samples contributed by other speakers to introduce variation in speaking style and pronunciation.

Each utterance is a short sentence (typically 6–12 words) recorded as a WAV audio file at 16 kHz. The utterances were constructed to cover a range of informal speech characteristics:

● Pure English informal speech: Grammatically standard English sentences with conversational register (e.g., "you do finish already or still doing", "the answer doesn't really make sense"). ● Manglish with discourse particles: English sentences augmented with Malay pragmatic particles (e.g., "i think lah this one better we do tomorrow", "this food not bad lah but a bit expensive"). ● Code-switched English–Malay: Sentences in which English and Malay words are interleaved (e.g., "this part susah sikit i need more time", "you boleh tolong explain this part tak"). ● Predominantly Malay sentences: Utterances with dominant Malay structure (e.g., "tadi lecturer explain laju sangat tak sempat nak faham", "saya rasa assignment ni susah sikit lah").

Each sample in the dataset contains:

● An audio file (WAV format) ● A reference English/code-switched transcription (reference) ● A Malay reference translation (reference_ms) annotated by the author

The reference transcriptions represent the ground-truth intended utterance, normalised to lowercase without punctuation for WER computation. The Malay reference translations represent the intended meaning expressed in standard Malay.

3.2.2Dataset Characteristics

The 61 utterances exhibit a variety of code-switching densities. Approximately one-third of utterances are predominantly English with discourse particles; approximately one-third involve moderate code-switching (at least two Malay content words embedded in an English sentence); and the remaining third range from heavily code-switched to predominantly Malay. This distribution reflects the natural code-switching continuum observed in Malaysian informal speech and ensures the evaluation covers a range of challenge levels.

Given the limited dataset size, a formal train and test split was not used, and evaluation was conducted on the full set of 61 utterances. The dataset includes recordings from multiple speakers, introducing a degree of speaker variability, although overall diversity remains limited. For the fine-tuning experiment, the same dataset was used for training. As such, the results should be interpreted as an analysis of model behaviour under constrained, low-resource conditions rather than a measure of generalisable performance.

3.2.3Text Normalisation

Before computing WER, all transcriptions both reference and model-predicted, undergo the following normalisation:

1. 1.Lowercasing
2. 2.Removal of punctuation (excluding apostrophes within words)
3. 3.Collapsing of multiple whitespace characters

This normalisation prevents mismatches due to capitalisation or punctuation differences from inflating WER scores.

3.3ASR Models and Configurations

Four ASR configurations were evaluated. All models were run on the same set of 61 audio files and scored against the same reference transcriptions.

3.3.1Whisper (Baseline)

OpenAI Whisper (model size: small, 39 million parameters) was used as the first baseline. The model was loaded via the openai-whisper Python library and applied in transcription mode with the language forced to English (language="en", task="transcribe"). This configuration was chosen to prevent Whisper from invoking its built-in translation mode, which would output English regardless of Malay content in the audio, and to ensure a fair comparison with the wav2vec models on the task of faithful transcription.

The Whisper small model employs a log-Mel spectrogram encoder and an autoregressive decoder. Its multilingual pretraining on 680,000 hours of diverse speech makes it substantially more robust to accented and informal speech than models trained solely on clean audiobook data.

3.3.2wav2vec 2.0 (Baseline)

The pretrained facebook/wav2vec2-base-960h model was used as the second baseline. This model uses a CTC decoder and produces token-level predictions, which are then decoded with greedy decoding (argmax over the output logits). The model was applied to audio resampled to 16 kHz, with stereo signals converted to mono by averaging channels.

This model represents the baseline capability of a strong English ASR model applied out-of-the-box to informal Malaysian speech, without any domain adaptation. It was trained exclusively on LibriSpeech, which consists of clean audiobook narration, making it the most domain-mismatched of the four configurations.

3.3.3wav2vec 2.0 (Fine-Tuned)

To investigate whether limited domain adaptation could improve recognition of Manglish and code-switched speech, a fine-tuning experiment was conducted on the facebook/wav2vec2-base-960h base model using the 61-utterance dataset.

Architecture adaptation: To avoid overfitting on a very small dataset and to preserve the pretrained acoustic representations, the majority of the model was frozen. Specifically, the entire wav2vec2 feature extractor and the lower encoder transformer layers were frozen during training. Only the last two encoder transformer layers and the final CTC projection head were left trainable. This selective unfreezing strategy is consistent with prior work on low-resource fine-tuning [3], [11], where training only the uppermost layers is found to reduce the risk of catastrophic forgetting while still allowing meaningful domain adaptation.

Training configuration: Supervised fine-tuning was performed using CTC loss. The audio was processed by the wav2vec2 processor to extract normalised input features, and the reference transcriptions were tokenised using the model's vocabulary. Labels corresponding to padding tokens were set to -100 to be ignored in the loss computation. The HuggingFace Trainer API was used with a batch size of 1 per device, 10 warmup steps, and fp16=False (full precision training).

Hyperparameter search: A systematic hyperparameter grid search was conducted to identify the best combination of learning rate and number of training epochs. Table 3.3.3:1 summarises the results.

Figure 3.3.3:2 presents the effect of both training epochs and learning rate on WER.

The top plot shows a clear overfitting pattern. Performance improves from 2 to 3 epochs, achieving the lowest WER of 0.700, before degrading significantly at 4 and 5 epochs. This indicates that excessive training on the small dataset leads to overfitting and loss of generalisation.

The bottom plot demonstrates that learning rate selection has a significant impact on model

performance. A high learning rate (1 × 10⁻⁵) results in severe degradation due to unstable updates, while very low learning rates provide minimal improvement. The optimal learning rate of 5 × 10⁻⁶ achieves the best balance between adaptation and stability.

The best configuration, learning rate 5 × 10⁻⁶ and 3 epochs, was selected for the final fine-tuned model and all downstream evaluations. The fine-tuned model was saved to disk and subsequently loaded for inference.

3.3.4wav2vec 2.0 + KenLM Language Model

A fourth configuration augmented the wav2vec 2.0 CTC decoder with an n-gram language model using the pyctcdecode library and a KenLM language model trained on standard English text. The LM serves to bias the beam search decoder towards more linguistically probable word sequences, theoretically improving transcription accuracy for well-formed English sentences.

The rationale for including this configuration was to test whether a standard English LM could help the wav2vec model produce cleaner, more coherent transcriptions of the English portions of code-switched utterances, even at the potential cost of filtering out Malay words. As discussed in the Results chapter, this hypothesis proved to be incorrect, and the LM integration led to significant performance degradation.

3.4Machine Translation Component

Translation from English/Manglish to Malay was performed using facebook/nllb-200-distilled-600M [14], a 600-million parameter distilled variant of the NLLB-200 multilingual translation model. The model supports 200 language pairs and is accessible through the HuggingFace Transformers library.

The translation target was set to Malay in Latin script (zsm_Latn) by forcing the beginning-of-sequence token to the corresponding language code. No fine-tuning was applied to the MT component. This design ensures that all observed differences in end-to-end BLEU scores across pipeline variants are attributable exclusively to variations in ASR quality, isolating the effect of ASR accuracy on translation performance.

The same MT model and configuration was applied to:

1. 1.Ground-truth transcriptions (to establish an upper-bound BLEU score)
2. 2.Whisper ASR outputs
3. 3.wav2vec baseline ASR outputs
4. 4.wav2vec fine-tuned ASR outputs
5. 5.wav2vec + KenLM ASR outputs

3.5Evaluation Metrics

3.5.1Word Error Rate (WER)

Word Error Rate is the standard evaluation metric for ASR systems. It quantifies the minimum edit distance between a predicted transcription and the reference at the word level, expressed as a fraction of the total number of words in the reference:

(3.1)

where S is the number of substitutions, D the number of deletions, I the number of insertions, and N the total number of reference words. A WER of 0.0 represents perfect transcription; a WER greater than 1.0 is possible when the number of insertions is large. WER was computed using the jiwer library, with text normalisation applied to both references and predictions before scoring.

3.5.2BLEU Score

Translation quality was evaluated using the Bilingual Evaluation Understudy (BLEU) score [16], computed with the sacrebleu library using its corpus_bleu function. BLEU measures n-gram overlap between the candidate translation and one or more human reference translations, incorporating a brevity penalty to discourage short translations:

(3.2)

where pn is the modified n-gram precision for order n, wn is the weighting factor (uniform, wn=1/N), and BP is the brevity penalty. BLEU scores range from 0 to 100; higher scores indicate closer correspondence to reference translations. Scores above 25 are generally considered to indicate good translation quality, while scores below 10 suggest poor quality.

3.5.3Upper-Bound BLEU

To contextualise pipeline BLEU scores, an upper-bound BLEU score was computed by passing the ground-truth English/Manglish reference transcriptions (rather than ASR predictions)

through the MT component. This represents the maximum achievable BLEU score given perfect ASR transcription, assuming the MT model performs ideally on the input text. The gap between the upper-bound BLEU and the pipeline BLEU scores quantifies the loss attributable to ASR errors.

3.5.4Qualitative Error Analysis

In addition to aggregate quantitative metrics, a qualitative error analysis was conducted by manually inspecting the ASR and MT outputs for all 61 utterances. Errors were categorised into recurring types: misrecognition of Manglish discourse particles, substitution of code-switched Malay words with phonetically similar English nonsense, complete omission of Malay-dominant segments, normalisation of Manglish to standard English by Whisper, and word-merging in the LM-augmented decoder. Representative examples are presented and discussed in Chapter 4.

3.6Implementation Details

3.6.1Development Environment

The system was implemented in Python using a modular script-based architecture. Each pipeline stage — ASR inference, fine-tuning, MT inference, and evaluation is implemented as a standalone script, allowing individual components to be executed and tested independently. The key libraries used are:

● openai-whisper: For loading and running the Whisper small model ● transformers (HuggingFace): For wav2vec 2.0 inference, fine-tuning, and NLLB-200 translation ● datasets (HuggingFace): For constructing the fine-tuning dataset from CSV ● torch / torchaudio: For tensor operations and audio resampling ● soundfile: For loading WAV audio files ● pyctcdecode: For KenLM-augmented beam search decoding ● jiwer: For WER computation ● sacrebleu: For BLEU score computation ● pandas: For dataset loading and result serialisation

All models were run on CPU. Audio files were stored as 16 kHz mono WAV files; stereo files were converted to mono by averaging channels prior to inference.

3.6.2Pipeline Execution Flow

The full pipeline executes in four sequential stages:

Stage 1 — ASR Inference. Each ASR configuration is run as a separate script over the 61 audio files. For Whisper, the whisper.load_model("small") call loads the pretrained model, and model.transcribe() is called per file with task="transcribe" and language="en" to force English-mode transcription. For wav2vec 2.0, the Wav2Vec2Processor and Wav2Vec2ForCTC are loaded from facebook/wav2vec2-base-960h; each audio file is preprocessed into normalised input features and passed through the model with torch.no_grad(), with greedy decoding (argmax over output logits) applied to produce the transcription. Each script writes its predictions to a CSV file in the results/ directory.

Stage 2 — Fine-Tuning. The fine-tuning script loads the wav2vec 2.0 base model and freezes all parameters in the wav2vec2 feature extractor and lower encoder layers via param.requires_grad = False. Only the last two encoder transformer layers (encoder.layers[-2:]) and the CTC projection head are left trainable. The 61-utterance dataset is loaded from CSV, audio is read with soundfile, and reference transcriptions are cleaned (lowercased, punctuation removed). The HuggingFace Trainer API handles the training loop with the configuration described in Section 3.3.3. The fine-tuned model is saved to disk and subsequently loaded for inference by a separate script.

Stage 3 — MT Inference. The MT script loads facebook/nllb-200-distilled-600M via AutoTokenizer and AutoModelForSeq2SeqLM. Translation is performed by tokenising the input text and calling model.generate() with forced_bos_token_id set to the zsm_Latn language token, directing the model to produce Malay in Latin script. The same function is applied to ground-truth transcriptions (upper-bound BLEU), Whisper outputs, wav2vec outputs, and fine-tuned outputs. All translations are written to a single consolidated CSV.

Stage 4 — Evaluation. WER is computed by a dedicated evaluation script using the jiwer library, applying the normalisation procedure described in Section 3.2.3 to both references and predictions before scoring. BLEU scores are computed using sacrebleu's corpus_bleu function against the Malay reference translations. Both WER and BLEU summaries are saved as CSV files for analysis.

3.6.3User Interface

A web-based user interface was implemented using Gradio, providing interactive access to the full pipeline without requiring command-line interaction.

The interface consists of two main components. The inference panel allows users to upload a WAV audio file and select an ASR model (Whisper or the fine-tuned wav2vec 2.0 model). The selected model transcribes the audio, and the resulting transcription is passed to the NLLB-200 model to generate a Malay translation. Both outputs are displayed in separate text fields. Figure

3.6.3:1 shows the main interface used for speech-to-text translation.

The evaluation panel presents key precomputed experimental results within the interface. This includes the per-category WER table (English-dominant, mixed, and Malay-dominant), the overall WER summary across ASR models, and BLEU score results. A bar chart visualising per-category WER is also included. A complete view of the evaluation panel is provided in Appendix A.

The interface runs locally in a web browser and demonstrates the practical usability of the system for informal Manglish speech input, while also providing a convenient way to inspect evaluation results.

4.1Overall ASR Performance

Table 4.1:1 presents the WER scores for all four ASR configurations evaluated on the 61-utterance informal Manglish dataset.

Figure 4.1:2 illustrates the relative performance of the evaluated ASR models in terms of Word Error Rate (WER). Whisper achieves the lowest WER of approximately 0.415, significantly outperforming all wav2vec 2.0 variants. The baseline wav2vec 2.0 model records a WER of 0.724, while fine-tuning provides only a modest improvement to 0.700 under the optimal configuration. The addition of a language model results in further degradation, indicating that it is not well-suited for the informal and code-switched nature of the dataset. Overall, the results demonstrate that pretrained multilingual models such as Whisper are more robust to real-world, informal speech conditions compared to models trained primarily on clean, monolingual data.

The superior performance of Whisper compared to wav2vec 2.0 can be attributed to differences in training data and model design. Whisper is trained on a large-scale multilingual and multi-domain dataset, which includes diverse accents, noisy environments, and informal speech patterns. This allows it to generalise better to real-world inputs such as Manglish and code-switched speech.

In contrast, wav2vec 2.0 is primarily trained on clean, monolingual English datasets such as

LibriSpeech. As a result, it is less robust to informal speech, pronunciation variation, and mixed-language input. This mismatch between training data and evaluation conditions leads to significantly higher error rates in the Manglish dataset.

To understand these aggregate results in more depth, the 61 utterances were assigned to the three categories defined in Section 3.2.2 English-dominant, mixed English–Malay, and Malay-dominant and WER was computed separately for each group. Table 4.1:3 presents the per-category WER for the three primary ASR configurations (KenLM is excluded as its pathological word-merging behaviour renders category-level WER uninformative).

Figure 4.1:4 shows the variation in WER across categories for the different ASR models.

Three findings emerge from this breakdown.

A consistent degradation gradient across categories. All three models show a clear progression from English-dominant (lowest WER) through mixed (moderate WER) to Malay-dominant (highest WER). This confirms that the challenge scales directly with Malay content density, and that the aggregate WER figures mask substantially different performance across utterance types.

Whisper's advantage is largest on English-dominant utterances. Whisper achieves a WER of 0.323 on English-dominant utterances compared to 0.589 for the wav2vec baseline, a gap of 26.6 percentage points. On mixed utterances the gap narrows to 22.7 points (0.528 vs. 0.755), and on Malay-dominant utterances both models effectively fail, with WERs of 1.184 and 1.211 respectively.

Fine-tuning helps mixed utterances but has no effect on Malay-dominant ones. The fine-tuned model reduces WER on mixed utterances from 0.755 to 0.721 (3.4 percentage points), and on English-dominant utterances from 0.589 to 0.573 (1.6 percentage points). However, for Malay-dominant utterances the fine-tuned WER is identical to the baseline (1.211), confirming that fine-tuning a model pretrained exclusively on English cannot address the fundamental Malay phonology gap. The model has learned to better handle Malay words embedded within English sentences, but remains unable to process utterances where Malay is the primary language. This finding refines the interpretation of the aggregate WER improvement (0.724 → 0.700) reported in Section 4.4: the gain is concentrated entirely in the English and mixed categories.

WER values exceeding 1.0 for Malay-dominant utterances indicate that the number of insertion errors produced by the models surpasses the total word count of the reference. This occurs because models encountering unfamiliar Malay phoneme sequences generate spurious English token sequences that bear no resemblance to the original utterance, producing outputs that are both substituted and inflated relative to the reference.

4.2Whisper vs. wav2vec: A Large Performance Gap

The most striking finding in the ASR results is the substantial gap between Whisper and the wav2vec configurations. Whisper achieves a WER of 0.415, compared to 0.724 for the wav2vec baseline, a difference of 30.9 percentage points. This gap is far larger than what would be expected from comparing these models on standard English benchmarks, where wav2vec 2.0-base-960h typically achieves WERs around 0.03–0.05 on LibriSpeech clean test sets.

Several factors explain this large discrepancy in the Manglish evaluation setting:

Multilingual pretraining: Whisper was trained on 680,000 hours of diverse, multilingual speech

including Malay. It therefore has direct exposure to Malay phonology and vocabulary, enabling it to recognise or approximate Malay words in code-switched utterances. In contrast, wav2vec 2.0-base-960h was trained exclusively on 960 hours of clean English audiobook speech and has no representation of Malay sounds in its vocabulary or training distribution.

Robustness to informal speech: Whisper's training data was collected from the internet, which includes informal, conversational, and accented speech. This gives Whisper a natural robustness advantage for informal Manglish speech compared to wav2vec trained on read speech.

Decoder architecture: Whisper's autoregressive sequence-to-sequence decoder incorporates an implicit language model that smooths the output and can leverage contextual information to recover plausible transcriptions. The wav2vec CTC decoder, by contrast, makes independent decisions at each time step and is more sensitive to acoustic ambiguity.

Example 1 illustrates the practical difference. For the utterance "you boleh tolong explain this part tak" (Can you help me explain this part?):

● Whisper: "you can help me explain this part, though." ● wav2vec baseline: "you bulito loge splendi spite da" ● wav2vec fine-tuned: "you boleh tolong explain this partak"

Whisper normalises the Malay code-switched words to their English translations ("boleh tolong" → "can help me"), which inflates its WER relative to the reference but produces a semantically reasonable transcription. The wav2vec baseline produces a completely unintelligible output. The fine-tuned model partially recovers the Malay words, which is discussed further in Section 4.4.

Example 2 illustrates discourse particle handling. For "eh actually i think this one can lah":

● Whisper: "actually, i think this one can laugh." (misinterprets "lah" as "laugh") ● wav2vec baseline: "e atoly i think the slankin la" (severe distortion) ● wav2vec fine-tuned: "e actually i think thisone canla" (improved but imperfect)

Whisper's tendency to reinterpret "lah" as "laugh" is a systematic error that occurs when the particle appears at the end of a sentence, exploiting the acoustic similarity between the terminal "lah" and the word "laugh."

4.3Comparison with Standard English and Code-Switching Benchmarks

To contextualise the performance of the evaluated ASR models, it is useful to compare the results obtained in this study with established benchmarks on both standard English speech and

code-switched speech.

On clean, monolingual English datasets such as LibriSpeech, wav2vec 2.0 achieves state-of-the-art performance, with reported Word Error Rates (WER) as low as 1.8% on the test-clean subset [13]. Whisper has also been shown to achieve low WER values on standard English benchmarks, typically within a few percentage points under controlled conditions [12]. These results indicate that both models are highly effective when evaluated in high-resource, standardised settings.

In contrast, the performance observed in this study is significantly worse. Whisper achieves a WER of 0.415, while wav2vec 2.0 records a WER of 0.724 on the informal Manglish dataset. This represents more than an order-of-magnitude degradation compared to standard English benchmarks.

The benchmark values are taken from prior LibriSpeech evaluations reported in the literature [12], [13].

Table 4.3:1 compares the WER obtained in this study with benchmark results reported in the literature.

This discrepancy highlights the impact of domain shift. Both models are trained primarily on clean, monolingual speech and therefore struggle when applied to informal, code-switched speech containing pronunciation variation, non-standard grammar, and mixed-language vocabulary.

A similar pattern has been observed in prior work on code-switched ASR. Yang et al. [7] report that Whisper experiences significant performance degradation on Mandarin–English code-switched speech compared to monolingual benchmarks. Likewise, Agro et al. [6] conclude in their systematic review that even large pretrained ASR models exhibit substantially higher WER when evaluated on code-switched input due to phonetic variability and language mixing.

The results of this study are consistent with these findings, but demonstrate an even more severe degradation. This is likely due to the additional complexity of Manglish, which combines code-switching with informal discourse particles, non-standard grammar, and reduced pronunciation. Unlike structured code-switching datasets used in prior studies, the speech in this dataset reflects natural conversational usage, making it particularly challenging for models trained on formal data.

Overall, this comparison reinforces the conclusion that strong performance on standard English benchmarks does not translate to real-world multilingual settings. The large gap between benchmark performance and Manglish performance underscores the need for domain-specific adaptation and more representative evaluation datasets.

4.4Fine-Tuning: Limited but Measurable Improvement

Fine-tuning wav2vec 2.0 on the 61-utterance Manglish dataset reduced WER from 0.724 to 0.700, an improvement of 2.4 percentage points. While modest, this improvement suggests some adaptation to informal speech patterns within the constraints of the dataset.

Given the small dataset size (61 utterances), this difference should be interpreted with caution, as small absolute changes in WER may not be statistically significant. A formal statistical significance test (e.g., bootstrap resampling) was not conducted due to time constraints, but would be required to confirm the reliability of the observed difference.

4.4.1Hyperparameter Analysis

The hyperparameter search (Table 3.3.3:1) reveals several important patterns:

Sensitivity to learning rate: A high learning rate of 1 × 10⁻⁵ catastrophically degrades performance (WER 0.950), causing the model to overfit aggressively to the small dataset and lose its pretrained English acoustic representations. A very low learning rate of 1 × 10⁻⁷ offers negligible improvement over baseline (0.726 vs. 0.724), as the update steps are too small to meaningfully adapt the model within three epochs. The optimal learning rate of 5 × 10⁻⁶ represents a sweet spot that allows the model to adapt to the informal domain without losing its pretrained capabilities.

Epoch-level overfitting: For the best learning rate (5 × 10⁻⁶), a clear overfitting pattern emerges: performance improves steadily from 2 epochs (WER 0.721) to 3 epochs (WER 0.700), but then degrades sharply at 4 epochs (WER 0.870) and deteriorates further at 5 epochs (WER 0.933). This U-shaped training curve is characteristic of fine-tuning a large pretrained model on very limited data: initial fine-tuning adapts the model to the domain, but continued training causes

the model to memorise specific training examples, losing generalisability. This pattern strongly reinforces the rationale for using a small number of epochs and supports the claim that the fine-tuning experiment was conducted under appropriately conservative settings.

Comparison with no-improvement baseline: At LR 1 × 10⁻⁶ , WER matches the untrained baseline exactly (0.724), suggesting that with too conservative a learning rate, the model effectively makes no meaningful updates within three epochs. This provides a useful null condition for interpreting the improvements observed at LR 5 × 10⁻⁶.

4.4.2Qualitative Improvement from Fine-Tuning

While the WER improvement is modest in aggregate, qualitative inspection reveals a specific and meaningful improvement in the fine-tuned model's handling of code-switched Malay words. For utterance 4 ("you boleh tolong explain this part tak"), the fine-tuned model produces "you boleh tolong explain this partak" — correctly recognising the Malay words "boleh tolong" that the baseline completely distorted. For utterance 7 ("eh actually i think this one can lah"), the fine-tuned model produces "e actually i think thisone canla", correctly preserving "actually," "think," and "can" with only minor distortions.

This improvement is attributable to the fine-tuned model having been exposed to the acoustic patterns of Malay words during training, allowing it to map Malay phonemes to their approximate phonetic representations. The base model, having never encountered Malay in training, attempts to map these unfamiliar phonemes to the most similar sounds in its English vocabulary.

4.4.3Limitations of the Fine-Tuning Approach

The fine-tuning experiment has a critical limitation that must be acknowledged: the same 61-sample dataset was used for both training and evaluation. This means the fine-tuned model is evaluated on data it was trained on, creating a risk that reported improvements reflect memorisation rather than generalisation. True generalisation can only be assessed using a held-out test set not seen during training. Given the dataset size of 61 samples, this was not feasible; even a modest test split of 20% would leave fewer than 50 training samples, likely insufficient for any meaningful adaptation.

The modest improvement observed (WER 0.700 vs. 0.724) is more consistent with limited domain adaptation than with pure memorisation of the training samples. If the model were memorising the 61 training utterances, WER on those same utterances would be expected to approach zero as training continued. Instead, WER improves modestly to 0.700 at 3 epochs and then degrades sharply to 0.870 at 4 epochs and 0.933 at 5 epochs. This in-sample degradation

at higher epoch counts is characteristic of catastrophic forgetting the model progressively overwrites pretrained acoustic representations under sustained weight updates rather than the convergence towards zero loss expected from memorisation. The 2.4 percentage point improvement at the optimal configuration is therefore unlikely to be explained by memorisation of fixed training examples alone.

That said, this reasoning provides only circumstantial evidence. Without a held-out test set, generalisation to unseen speech cannot be confirmed. The results should be interpreted as indicative of domain adaptation effects under constrained, low-resource conditions rather than as evidence of robust generalised improvement.

4.5Language Model Integration: Degraded Performance

The KenLM-augmented decoder yields the worst performance of all four configurations, with a WER of 0.916. This result is unexpected from the perspective of standard ASR theory, where LM integration typically improves transcription accuracy by constraining the output to grammatically plausible sequences.

4.5.1Word-Merging Behaviour

Inspection of the wav2vec + LM outputs reveals a critical failure mode: the decoder consistently merges multiple words into single concatenated strings without whitespace, producing outputs such as:

● Reference: "i think lah this one better we do tomorrow" ● LM output: "itingladisone beteittomoro" ● Reference: "why you never reply my message just now" ● LM output: "why youneverweblam ymaysisjust now"

This word-merging behaviour indicates that the LM beam search decoder lost track of word boundaries during decoding. The most probable explanation is a configuration mismatch between the KenLM vocabulary and the CTC output token space, causing the decoder to collapse multiple tokens into single LM vocabulary entries. This results in outputs that are, from a word-level perspective, effectively a single long non-word token, dramatically increasing the number of insertion and substitution errors recorded by WER.

4.5.2Why LM Integration Failed

Several factors contributed to this failure:

Vocabulary mismatch: The KenLM model was trained on standard English text and does not

include Malay words or Manglish particles in its vocabulary. When the beam search encounters acoustic evidence for a Malay word, the LM assigns it near-zero probability, biasing the decoder away from correct Malay transcriptions and towards whatever English sequence best fits the acoustic signal — often at the cost of word boundary integrity.

Domain mismatch: Even for English words, the informal register and non-standard grammar of Manglish utterances would be assigned low probability by a standard English LM, potentially pushing the decoder towards unlikely but more grammatically regular alternatives.

Beam search configuration: The complete merging of words suggests that the decoding configuration (word separator tokens, LM weight, beta parameter) was not correctly tuned for the Manglish domain. A dedicated KenLM model trained on Manglish or Malaysian English text, with appropriately tuned decoding hyperparameters, might yield different results.

The degraded performance of the wav2vec + KenLM configuration is partly attributable to suboptimal decoder parameter settings (e.g., LM weight and word boundary handling), which were not exhaustively tuned within the project scope. As such, the results should be interpreted as indicative of integration challenges rather than the inherent limitations of language model fusion.

This finding illustrates a broader principle: adding a language model to an ASR system does not automatically improve performance; the LM must be trained on data from the same domain and language variety as the target speech. For code-switched Manglish, this requires a specialised LM incorporating both English and Malay vocabulary with appropriate code-switching patterns.

4.6Machine Translation Results

4.6.1Upper-Bound BLEU

When ground-truth English/Manglish transcriptions are passed through the NLLB-200 MT model, the corpus BLEU score on the Malay reference translations is 26.78. This serves as the upper-bound BLEU — the best the MT component can achieve assuming perfect ASR transcription.

A BLEU score of 26.78 is in the range generally described as "understandable to good" translation quality. The fact that the upper bound falls below 40 (typically considered "high quality") reflects the inherent challenge that the MT model faces with Manglish input. Even given perfect transcription, the Manglish sentences contain code-switched words, discourse particles, and non-standard grammar that the NLLB-200 model — trained on formal text — does

not handle perfectly.

Qualitative inspection of the ground-truth MT outputs identifies two sources of loss that explain why the upper bound falls below 40.

Manglish vocabulary misinterpretation. NLLB-200 was trained on formal parallel text and does not recognise Manglish words used with informal meanings. For example:

● Reference: "kalau awak free kita boleh jumpa sekejap" ● Reference Malay: "kalau awak free kita boleh jumpa sekejap" ● Ground-truth MT: "Kalau awak dah bangun kita boleh jumpa sekejap"

The Manglish word "free" (meaning available, not busy) is misread by the MT model, which produces "dah bangun" (already awake/woken up) — a semantically wrong translation despite perfect transcription. The intended meaning ("if you're free, we can meet briefly") is lost.

Non-standard Manglish grammar. Manglish affirmative constructions do not map cleanly to standard Malay. For utterance 11 ("like that also can no problem"):

● Ground-truth MT: "seperti itu juga tidak boleh masalah" ● Reference Malay: "macam itu pun boleh tiada masalah"

The MT model renders "also can" as "tidak boleh" (cannot), inverting the polarity of the sentence. The intended affirmative meaning is reversed.

Discourse particles, by contrast, are handled with variable success. For utterance 7 ("eh actually i think this one can lah"):

● Reference: "eh actually i think this one can lah" ● Ground-truth MT: "eh sebenarnya saya fikir yang ini boleh lah"

Here the MT preserves "lah" faithfully as "boleh lah" — an acceptable informal Malay rendering. The particle is not mishandled when passed as clean text. This stands in direct contrast to the Whisper pipeline result for the same utterance: Whisper transcribes "lah" as "laugh," and the MT faithfully translates "laugh" to "ketawa," producing "Sebenarnya, saya rasa yang ini boleh ketawa" (can laugh). This error originates entirely in the ASR stage, not in the MT model.

This distinction matters for interpreting the upper-bound BLEU. The gap between the upper bound (26.78) and perfect translation quality reflects genuine MT difficulty with informal Manglish vocabulary and grammar structures. The further gap between the upper bound and the pipeline BLEU scores (9.64 for Whisper, ~2.3 for wav2vec) reflects compounded ASR error

propagation layered on top of these intrinsic MT limitations.

4.6.2End-to-End Pipeline BLEU Results

Table 4.6.2:1 presents the end-to-end BLEU scores for all pipeline configurations.

The wav2vec + KenLM configuration achieves a BLEU score of 2.05, which is slightly lower than both the baseline wav2vec (2.35) and the fine-tuned model (2.28). Despite the use of a language model, no improvement is observed, indicating that the integration does not enhance downstream translation performance.

The relatively small reduction in BLEU compared to the large increase in WER (0.916 vs 0.724 for the baseline) suggests a floor effect. At this level of ASR error, translation quality is already extremely low for all wav2vec-based pipelines, and further degradation in transcription does not result in a proportionate decrease in BLEU. This indicates that the MT model is already operating on severely corrupted input across these configurations.

As shown in Figure 4.6.2:2, there is a significant drop in BLEU score from clean input to ASR-generated input, clearly demonstrating the cascading error effect.

The results reveal three key findings. First, there is a substantial gap between the upper bound BLEU (26.78) and all ASR-based pipeline outputs, highlighting the impact of transcription errors on translation quality. Second, the Whisper pipeline significantly outperforms wav2vec-based approaches, achieving a BLEU score of 9.64 compared to approximately 2.3 for wav2vec variants. Third, neither fine-tuning nor language model integration leads to meaningful improvements in translation quality, as the BLEU scores remain consistently low across wav2vec configurations.

4.6.3The Cascading Error Effect

The most critical finding of this study is the dramatic deterioration from upper-bound BLEU (26.78) to pipeline BLEU scores. The Whisper pipeline loses 17.14 BLEU points compared to the upper bound, representing a 64% reduction. The wav2vec pipelines are near-zero, representing effectively unusable translation outputs.

This degradation is a direct consequence of cascading errors. When ASR errors introduce unfamiliar tokens into the MT input — whether phonetically distorted English words, randomly assorted consonant clusters, or merged strings — the MT model cannot recover the intended meaning. The translation model, trained on clean text, maps these tokens to whatever Malay sequences have the highest likelihood given the corrupted input, producing semantically incoherent translations.

The following examples from the full results CSV illustrate this mechanism:

Example A (moderate ASR error, partial recovery):

● Reference: "i think lah this one better we do tomorrow"

● Reference Malay: "saya rasa yang ini lebih baik kita buat esok" ● Whisper prediction: "i think la, this one better we do tomorrow." ● Whisper MT: "Saya rasa La, ini lebih baik kita lakukan esok." ● Assessment: Near-correct Malay translation, only the untranslated particle "La" is out of place.

Example B (severe ASR error, total translation failure):

● Reference: "this part susah sikit i need more time" ● Reference Malay: "bahagian ini agak susah saya perlukan lebih masa" ● wav2vec prediction: "despart sussasci kid and eat morth nine" ● wav2vec MT: "Berlepas Susasci anak dan makan Mort Nine" ● Assessment: The ASR output is entirely unintelligible; the MT model generates a Malay sentence mapping "kid" to "anak" (child), "eat" to "makan" (eat), and hallucinating the phrase "Berlepas Susasci" — a meaningless string.

Example C (Whisper normalisation behaviour):

● Reference: "you boleh tolong explain this part tak" ● Reference Malay: "awak boleh tolong jelaskan bahagian ini tak" ● Whisper: "you can help me explain this part, though." ● Whisper MT: "boleh tolong saya jelaskan bahagian ini, walaupun." ● Assessment: Whisper translates "tak" as "though" and paraphrases "boleh tolong" as "can help me"; the MT output is semantically plausible but does not match the reference closely. BLEU is reduced.

Example D (predominantly Malay input, near-total failure):

● Reference: "tadi lecturer explain laju sangat tak sempat nak faham" ● Reference Malay: "tadi pensyarah jelaskan terlalu laju tak sempat faham" ● Whisper: "now" ● Whisper MT: "Sekarang" ● Assessment: Whisper recognises only the word "tadi" (just now) and maps it to "now," completely losing the rest of the utterance. The MT produces a one-word translation.

Example D is particularly instructive. When an utterance is predominantly Malay, even Whisper, the strongest ASR model, fails almost completely. Whisper's English-forced transcription mode causes it to map the Malay speech to a minimal English sequence rather than faithfully representing the mixed-language content. This confirms that the "language=en" setting in Whisper penalises Malay-dominant speech and suggests that a Malay-capable or

language-agnostic transcription mode might yield better results.

4.6.4Why Fine-Tuning Improves WER but Not BLEU

The observation that fine-tuning improves WER (from 0.724 to 0.700) but does not improve end-to-end BLEU (2.35 → 2.28) is an important finding that requires careful interpretation.

The key insight is that WER and BLEU measure different aspects of quality, and their relationship is highly non-linear in the presence of severe errors. A WER of 0.724 already implies that almost three out of every four words in the average transcription are wrong. At this level of error, the ASR output is largely unintelligible to the downstream MT model. Reducing WER to 0.700 means that the fine-tuned model makes slightly fewer errors per utterance, but the transcription is still sufficiently corrupted that the MT model cannot produce coherent translations.

In other words, there is an error floor below which WER improvements do not translate into BLEU improvements, because the MT model requires a minimum threshold of recognisable input tokens to produce a meaningful translation. The fine-tuning experiment did not reduce WER sufficiently to cross this threshold. This finding is consistent with prior work [5], [6], which shows that once ASR error rates exceed moderate levels, translation quality deteriorates rapidly, as MT systems fail to recover meaning from corrupted input sequences. Small improvements in WER at high error rates therefore have negligible impact on translation quality.

Additionally, the fine-tuned model's qualitative improvement, better recognition of Malay words, does not necessarily help the NLLB-200 MT model, which is designed to translate from English (or clean Malay) to Malay. Producing a more accurate Manglish transcription (e.g., "you boleh tolong explain this partak") may actually be harder for the MT model to handle than Whisper's normalised English equivalent ("you can help me explain this part"), since the MT model was not trained on code-switched input.

Furthermore, not all transcription errors contribute equally to translation performance. Fine-tuning may reduce minor errors such as function word substitutions or small phonetic distortions, but still fail to correctly recognise key content words that carry the core meaning of the sentence. As a result, even with lower WER, the semantic quality of the transcription may not improve sufficiently to benefit the downstream translation model. This highlights the limitation of using WER alone as a proxy for end-to-end system performance in cascaded pipelines.

4.7Summary of Qualitative Error Patterns

Manual inspection of all 61 utterances identified the following recurring error types:

4.7.1Manglish Discourse Particles

Particles such as lah, ah, kan, wah, and sumpah are systematically mishandled by all models:

● Whisper maps "lah" to "la", "laugh," or "la" (comma-separated), or drops it entirely. ● wav2vec maps these particles to phonetically similar English syllables ("la" → "letn", "sumpah" → "soopa"). ● Fine-tuning partially improves particle recognition in isolation but does not handle all cases. ● None of the models pass discourse particles faithfully into the MT input in a form the MT model can use.

4.7.2Code-Switched Malay Content Words

Malay content words embedded in English sentences present the greatest challenge for wav2vec:

● "susah sikit" (a little difficult) → wav2vec: "sussasci kid"; Whisper: "susasi kid" ● "boleh" (can/able) → wav2vec baseline: "bulito"; fine-tuned: "boleh" (correct) ● "tolong" (help) → wav2vec baseline: "loge"; fine-tuned: "tolong" (correct)

The fine-tuned model shows substantially better handling of common Malay function words ("boleh", "tolong"), which is the source of the WER improvement. However, for less common Malay words or fully Malay-dominant sentences, even the fine-tuned model fails.

4.7.3Whisper's Normalisation Tendency

Whisper exhibits a systematic tendency to normalise code-switched utterances to standard English:

● "maybe kita boleh buat cara lain" → "maybe we can do it another way." ● "kalau awak free kita boleh jumpa sekejap" → "if it's not fit we can try it for a while" ● "i try already but still tak jalan" → "i tried already but still not"

This normalisation behaviour reduces WER compared to wav2vec (because standard English words match the reference better than phonetic distortions) but distorts the meaning of the utterance. Notably, "tak jalan" (not working/doesn't work) becomes simply "not," losing the

idiomatic meaning of the expression. When such normalised outputs are passed to the MT model, the semantic information encoded in the Malay words is lost.

4.7.4Predominantly Malay Utterances

Utterances with a dominant Malay structure (e.g., "tadi lecturer explain laju sangat tak sempat nak faham", "kalau awak free kita boleh jumpa sekejap") produce catastrophically low-quality transcriptions across all models:

● Whisper collapses these to a single English word or short phrase ● wav2vec produces random phonetic sequences with no resemblance to either language ● Fine-tuning shows minimal improvement

This finding suggests that the 61-utterance dataset's most challenging samples — those that are predominantly Malay — represent a qualitatively different problem that cannot be addressed by fine-tuning a model that was never exposed to Malay in its pretraining.

4.7.5Error Propagation into Translation

All of the above ASR error patterns propagate into the MT stage and are amplified. The MT model's responses to corrupted input include:

● Hallucination: The model generates plausible-sounding Malay text that has no semantic connection to the original utterance (e.g., "Berlepas Susasci anak dan makan Mort Nine" for "this part is a little difficult, I need more time"). ● Literal mapping of nonsense: The model translates recognisable but misused English words literally, producing grammatically correct but semantically wrong Malay (e.g., "can laugh" → "boleh ketawa" for a sentence that should mean "can do"). ● Partial recovery: For Whisper outputs where the normalisation is semantically close to the reference, the MT model can produce a reasonable Malay translation, contributing to Whisper's relatively higher BLEU of 9.64.

4.8Discussion: Implications for System Design

The results of this evaluation carry several practical implications for the design of speech translation systems in informal multilingual settings:

ASR model selection is the dominant factor: The choice between Whisper and wav2vec has a larger impact on end-to-end translation quality than any amount of fine-tuning achievable under low-resource constraints. For practical deployment in a Manglish speech translation application, a Whisper-based system would be substantially more useful than any wav2vec

variant evaluated here.

BLEU as an aggregate metric has limitations: The aggregate BLEU score masks important qualitative differences between pipeline configurations. A BLEU score of 9.64 for the Whisper pipeline represents genuinely useful translations for English-dominant utterances (where Whisper performs well) mixed with very poor translations for Malay-dominant utterances (where Whisper collapses to near-zero output). This bimodal behaviour is not visible in the aggregate score.

Translation quality is bottlenecked by ASR: Given the upper-bound BLEU of 26.78, even with perfect ASR the MT component faces challenges with Manglish input. However, the gap between 26.78 and 9.64 — representing the loss attributable to ASR errors — dwarfs the MT model's intrinsic limitations and confirms that improving ASR accuracy is the highest-leverage intervention for improving end-to-end translation quality.

LM integration requires domain-specific adaptation: The failure of the generic English KenLM demonstrates that language model integration for code-switched ASR requires a code-switching-aware LM trained on representative Manglish data. Generic monolingual LMs are likely to harm rather than help performance in this setting.

5.1Summary of Findings

This project designed and evaluated a cascaded speech-to-text translation pipeline for informal Malaysian English (Manglish) and English–Malay code-switched speech, addressing a significant gap in the evaluation of existing ASR and MT technologies under realistic, informal multilingual conditions.

The key findings are:

1. 1.Whisper substantially outperforms wav2vec 2.0 on informal Manglish speech (WER 0.415 vs. 0.724), primarily because Whisper's multilingual pretraining gives it exposure to Malay phonology and its training on informal web-collected audio makes it more robust to conversational speech.
2. 2.Limited fine-tuning of wav2vec 2.0 yields a small but consistent, though not statistically confirmed, WER improvement (0.724 → 0.700), with the best configuration using a learning rate of 5 × 10⁻⁶ and 3 training epochs. The hyperparameter analysis reveals clear overfitting at higher learning rates and more epochs, consistent with the constraints of the 61-sample dataset.
3. 3.WER improvement from fine-tuning does not translate into BLEU improvement (2.35 → 2.28), because the wav2vec error rate remains well above the threshold required for coherent MT input. This finding empirically demonstrates the non-linear relationship between ASR accuracy and downstream translation quality.
4. 4.KenLM language model integration significantly degrades performance (WER 0.916), producing pathological word-merging behaviour attributable to vocabulary mismatch between the English LM and the Manglish test domain.
5. 5.The cascading error effect is severe: End-to-end pipeline BLEU scores (9.64 for Whisper, ~2.3 for wav2vec) fall dramatically short of the upper-bound BLEU (26.78), confirming that ASR errors compound at the translation stage. Even the MT component itself struggles with Manglish input given perfect transcription, with the upper-bound falling below 30.
6. 6.Qualitative analysis identifies Manglish discourse particles, code-switched Malay words, and predominantly Malay utterances as the primary failure modes across all models.

5.2Contributions

The specific contributions of this project are:

1. 1.A manually constructed evaluation dataset of 61 informal, code-switched English–Malay utterances with both English reference transcriptions and Malay reference translations, representing a domain not covered by any existing publicly available benchmark.
2. 2.A comparative evaluation of four ASR configurations under identical conditions on informal Manglish speech, providing a systematic analysis of model-level and configuration-level differences in a domain where prior empirical data is sparse.
3. 3.Empirical evidence of the WER–BLEU decoupling phenomenon in a code-switched, low-resource setting, demonstrating that fine-tuning can improve transcription accuracy while having negligible impact on translation quality when baseline WER is very high.
4. 4.Characterisation of LM integration failure modes in a code-switched domain, contributing to understanding of why standard LM fusion approaches fail outside their training distribution.
5. 5.A complete pipeline implementation demonstrating the integration of Whisper, wav2vec 2.0 (with and without fine-tuning), KenLM decoding, and NLLB-200 translation in a reproducible Python codebase.
6. 6.A structured qualitative error taxonomy for Manglish ASR and MT errors, providing an analytical framework for understanding system failure patterns in informal multilingual speech.

5.3Limitations

Several limitations should be noted when interpreting the results:

Dataset size and no train/test split: The dataset consists of 61 utterances, which is small by standard speech recognition benchmarks. As publicly available datasets do not capture informal Manglish and English–Malay code-switching, a bespoke dataset was constructed. The dataset includes recordings from multiple speakers, to introduce some variation, although overall speaker diversity remains limited.

Due to the dataset size, the same data was used for both fine-tuning and evaluation, which limits the ability to assess generalisation. The hyperparameter overfitting pattern observed in Table 3.3.3:1, where WER degrades sharply beyond 3 epochs even on the training data, provides indirect evidence that the improvements are not solely due to memorisation.

As such, this work focuses on analysing model behaviour under constrained, low-resource conditions rather than making generalisable performance claims. A larger, multi-speaker dataset with a proper train and test split is identified as future work.

Single MT model and no MT fine-tuning: Using a single, unfine-tuned MT model means that the MT component's limitations are conflated with ASR limitations in the BLEU evaluation. A dedicated English–Manglish to Malay MT model, if one existed, might produce substantially different results.

No Malay-to-English evaluation: While the interim report specified bidirectional translation as an objective, the evaluation focuses on English-to-Malay. Malay-to-English evaluation would require ASR models capable of handling Malay as the primary language, which is beyond the current system's capability.

Whisper forced-English mode: Using language="en" in Whisper forces the model to interpret all speech as English, which is appropriate for Manglish evaluation but artificially penalises predominantly Malay utterances. A language-agnostic or Malay-capable setting might perform better on the most challenging samples.

LM hyperparameter tuning: The KenLM integration was not extensively tuned (e.g., LM weight, beam width, and word boundary handling), and the observed degradation should therefore be interpreted as a configuration limitation rather than an inherent failure of language model integration.

In addition, this behaviour can be attributed to domain mismatch between the language model and the evaluation data. The KenLM model is trained on formal, standard English text, whereas the dataset in this study consists of informal, code-switched Manglish speech. As a result, the language model biases the decoding process towards standard English sequences, which can override the acoustic signal and suppress correct code-switched words. This mismatch is particularly problematic in Manglish, where non-standard grammar and mixed-language expressions are common.

Taken together, these findings suggest that effective language model integration in this context would require both domain-specific training data and careful decoder parameter tuning.

5.4Future Work

Based on the findings and limitations of this project, several directions for future research are identified:

1. 1.Larger and more diverse dataset: The most impactful improvement would be the construction or curation of a larger code-switched Manglish speech dataset with multiple speakers, diverse topics, and a proper train/test split. Datasets such as CS-FLEURS [10] provide a starting point but do not capture natural Malaysian conversational code-switching.
2. 2.Whisper fine-tuning for Manglish: Whisper's superior baseline performance suggests it is the more promising model for domain adaptation. Fine-tuning Whisper (or a Whisper-based variant) on Manglish data — even at a small scale — could potentially yield meaningful improvements, particularly for discourse particle handling and predominantly Malay utterances. Recent work by Yang et al. [7] and Anidjar et al. [11] provides methodological guidance for this approach.
3. 3.Language-agnostic or multilingual ASR mode: Instead of forcing Whisper to transcribe in English, a language-agnostic mode that can handle both English and Malay phonemes within a single utterance would be more appropriate for code-switched speech. Alternatively, a dedicated code-switching ASR model trained on bilingual English–Malay speech data would directly address the problem.
4. 4.NLLB-200 fine-tuning for Manglish input: Fine-tuning the MT component on code-switched English–Malay input–output pairs (if such data can be assembled) would improve the MT model's ability to handle Manglish grammar, discourse particles, and partial code-switching. This would increase the upper-bound BLEU and make the translation component more robust to the kinds of informal input it receives in a real pipeline.
5. 5.Domain-specific KenLM: Training a KenLM model on Manglish text — incorporating both English and Malay vocabulary with appropriate code-switching transitions — and properly tuning the beam search parameters would address the core failure mode identified in the LM experiment. Online Malaysian English text resources (social media, online forums) would provide suitable training material.
6. 6.End-to-end joint ASR–MT models: Emerging end-to-end speech translation architectures that jointly optimise ASR and MT in a single model [6] offer a promising alternative to the cascaded pipeline. By eliminating the hard boundary between recognition and translation, such models can potentially avoid the cascading error amplification observed in this study. Evaluating such models in the Manglish domain is a valuable direction for future work.
7. 7.Evaluation of commercial ASR systems: Commercial ASR APIs (Google Cloud Speech-to-Text, Microsoft Azure, etc.) may have proprietary training data including Malaysian English and could perform substantially better than the open-source models evaluated here. A comparative evaluation would establish the performance gap between open-source and commercial solutions in this domain.

1. [1]G. I. Winata, A. F. Aji, Z.-X. Yong, and T. Solorio, "The decades progress on code-switching research in NLP: A systematic survey on trends and challenges," in Proc. Findings of the Assoc. Comput. Linguistics (ACL), 2023, pp. 1–28.
2. [2]T. Nguyen and H.-D. Tran, "AsyncSwitch: Asynchronous text–speech adaptation for code-switched ASR," arXiv:2506.14190, 2025.
3. [3]S. Bansal, H. Kamper, K. Livescu, A. Lopez, and S. Goldwater, "Low-resource speech-to-text translation," in Proc. Interspeech, Hyderabad, India, 2018, pp. 1298–1302.
4. [4]T.-P. Tan, L. Qin, S. F. S. Juan, and J. Y. M. Khaw, "Low-resource Malay dialect automatic speech recognition modeling using transfer learning from a standard Malay model," Pertanika J. Sci. Technol., vol. 32, no. 1, pp. 45–62, 2024.
5. [5]Y.-T. Phua, S. Navaratnam, C.-M. Kang, and W.-S. Chew, "Sequence-to-sequence neural machine translation for English–Malay," Int. J. Artif. Intell., vol. 10, no. 2, pp. 85–97, 2022.
6. [6]M. T. Agro, A. Kulkarni, K. Kadaoui, Z. Talat, and H. Aldarmaki, "Code-switching in end-to-end automatic speech recognition: A systematic literature review," arXiv:2507.07741, 2025.
7. [7]Y. Yang, Y. Peng, X. Zhong, H. Huang, and E. S. Chng, "Adapting OpenAI's Whisper for speech recognition on code-switch Mandarin–English speech," arXiv:2311.17382, 2023.
8. [8]A. Adila, D. Lestari, A. Purwarianti, D. Tanaya, K. Azizah, and S. Sakti, "Enhancing Indonesian automatic speech recognition: Evaluating multilingual models with diverse speech variabilities," arXiv:2410.08828, 2024.
9. [9]J. Treffers-Daller, S. Majid, Y. N. Thai, and N. Flynn, "Explaining the diversity in Malay–English code-switching patterns: The contribution of typological similarity and bilingual optimization strategies," Languages, vol. 7, no. 4, art. no. 299, 2022.
10. [10]B. Yan et al., "CS-FLEURS: A massively multilingual and code-switched speech dataset," arXiv:2509.14161, 2025.
11. [11]R. Anidjar, A. Baevski, M. A. Hasegawa-Johnson, and E. Dupoux, "Whisper turns stronger: Augmenting wav2vec 2.0 for low-resource languages," arXiv:2501.00425, 2025.
12. [12]A. Radford et al., "Robust speech recognition via large-scale weak supervision," in Proc. Int. Conf. Mach. Learn. (ICML), 2023.
13. [13]A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," in Adv. Neural Inf. Process. Syst. (NeurIPS), 2020.
14. [14]NLLB Team et al., "No language left behind: Scaling human-centered machine translation," arXiv:2207.04672, 2022.
15. [15]K. Heafield, "KenLM: Faster and smaller language model queries," in Proc. Workshop Stat. Mach. Transl., 2011, pp. 187–197.
16. [16]K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "BLEU: A method for automatic evaluation of machine translation," in Proc. Annu. Meet. Assoc. Comput. Linguistics (ACL), 2002, pp. 311–318.
17. [17]S. Sitaram, V. Choudhary, S. Bali, and M. Choudhury, "Code-switched speech recognition: A survey," ACM Computing Surveys, vol. 52, no. 4, 2019.
18. [18]M. Sperber, G. Neubig, J. Niehues, and A. Waibel, "Attention-passing models for robust and data-efficient end-to-end speech translation," Transactions of the Association for Computational Linguistics, vol. 7, pp. 313–325, 2019.

Appendix A

Full Evaluation Panel

This appendix presents the complete view of the system interface, including the inference panel and the evaluation panel displaying precomputed experimental results.