AdaMix, a parameter-efficient fine-tuning method, outperforms full model fine-tuning in few-shot NLU tasks across benchmarks like GLUE. Using prompt-based strategies without extra validation or unlabeled data, AdaMix consistently boosts performance with both BERT and RoBERTa encoders, demonstrating stability and efficiency in few-shot scenarios.AdaMix, a parameter-efficient fine-tuning method, outperforms full model fine-tuning in few-shot NLU tasks across benchmarks like GLUE. Using prompt-based strategies without extra validation or unlabeled data, AdaMix consistently boosts performance with both BERT and RoBERTa encoders, demonstrating stability and efficiency in few-shot scenarios.

Smarter AI Training with Few-Shot Natural Language Tasks

2025/10/02 17:00

Abstract and 1. Introduction

  1. Background

    2.1 Mixture-of-Experts

    2.2 Adapters

  2. Mixture-of-Adaptations

    3.1 Routing Policy

    3.2 Consistency regularization

    3.3 Adaptation module merging and 3.4 Adaptation module sharing

    3.5 Connection to Bayesian Neural Networks and Model Ensembling

  3. Experiments

    4.1 Experimental Setup

    4.2 Key Results

    4.3 Ablation Study

  4. Related Work

  5. Conclusions

  6. Limitations

  7. Acknowledgment and References

Appendix

A. Few-shot NLU Datasets B. Ablation Study C. Detailed Results on NLU Tasks D. Hyper-parameter

A Few-shot NLU Datasets

Data. In contrast to the fully supervised setting in the above experiments, we also perform fewshot experiments following the prior study (Wang et al., 2021) on six tasks including MNLI (Williams et al., 2018), RTE (Dagan et al., 2005; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), QQP[1] and SST-2 (Socher et al.). The results are reported on their development set following (Zhang et al., 2021). MPQA (Wiebe et al., 2005) and Subj (Pang and Lee, 2004) are used for polarity and subjectivity detection, where we follow (Gao et al., 2021) to keep 2, 000 examples for testing. The few-shot model only has access to |K| labeled samples for any task. Following true few-shot learning setting (Perez et al., 2021; Wang et al., 2021), we do not use any additional validation set for any hyper-parameter tuning or early stopping. The performance of each model is reported after fixed number of training epochs. For a fair comparison, we use the same set of few-shot labeled instances for training as in (Wang et al., 2021). We train each model with 5 different seeds and report average performance with standard deviation across the runs. In the few-shot experiments, we follow (Wang et al., 2021) to train AdaMix via the prompt-based fine-tuning strategy. In contrast to (Wang et al., 2021), we do not use any unlabeled data.

\

B Ablation Study

\ Table 11: Ablation study demonstrating the impact of parameter sharing in AdaMix adapter framework.

\

C Detailed Results on NLU Tasks

The results on NLU tasks are included in Table 1 and Table 13. The performance AdaMix with RoBERTa-large encoder achieves the best performance in terms of different task metrics in the GLUE benchmark. AdaMix with adapters is the

\ \ Table 12: Varying the bottleneck dimension of adapters in AdaMix with BERT-base and RoBERTa-large encoder. * denotes the bottleneck dimension used in AdaMix with adapters.

\ \ only PEFT method which outperforms full model fine-tuning on all the tasks and on average score. Additionally, the improvement brought by AdaMix is more significant with BERT-base as the encoder, demonstrating 2.2% and 1.2% improvement over the performance of full model fine-tuning and the best performing baseline UNIPELT with BERTbase. The improvement is observed to be consistent as that with RoBERTa-large on every task. The NLG results are included in Table 4 and 5.

D Hyper-parameter

Detailed hyper-parameter configuration for different tasks presented in Table 15 and Table 16.

\

:::info Authors:

(1) Yaqing Wang, Purdue University ([email protected]);

(2) Sahaj Agarwal, Microsoft ([email protected]);

(3) Subhabrata Mukherjee, Microsoft Research ([email protected]);

(4) Xiaodong Liu, Microsoft Research ([email protected]);

(5) Jing Gao, Purdue University ([email protected]);

(6) Ahmed Hassan Awadallah, Microsoft Research ([email protected]);

(7) Jianfeng Gao, Microsoft Research ([email protected]).

:::


:::info This paper is available on arxiv under CC BY 4.0 DEED license.

:::

[1] https://www.quora.com/q/quoradata/

Piyasa Fırsatı
Sleepless AI Logosu
Sleepless AI Fiyatı(AI)
$0.03693
$0.03693$0.03693
-1.20%
USD
Sleepless AI (AI) Canlı Fiyat Grafiği
Sorumluluk Reddi: Bu sitede yeniden yayınlanan makaleler, halka açık platformlardan alınmıştır ve yalnızca bilgilendirme amaçlıdır. MEXC'nin görüşlerini yansıtmayabilir. Tüm hakları telif sahiplerine aittir. Herhangi bir içeriğin üçüncü taraf haklarını ihlal ettiğini düşünüyorsanız, kaldırılması için lütfen [email protected] ile iletişime geçin. MEXC, içeriğin doğruluğu, eksiksizliği veya güncelliği konusunda hiçbir garanti vermez ve sağlanan bilgilere dayalı olarak alınan herhangi bir eylemden sorumlu değildir. İçerik, finansal, yasal veya diğer profesyonel tavsiye niteliğinde değildir ve MEXC tarafından bir tavsiye veya onay olarak değerlendirilmemelidir.

Ayrıca Şunları da Beğenebilirsiniz

US SEC approves options tied to Grayscale Digital Large Cap Fund and Cboe Bitcoin US ETF Index

US SEC approves options tied to Grayscale Digital Large Cap Fund and Cboe Bitcoin US ETF Index

PANews reported on September 18th that the U.S. Securities and Exchange Commission (SEC) announced that, in addition to approving universal listing standards for commodity-based trust units , the SEC has also approved the listing and trading of the Grayscale Digital Large Cap Fund, which holds spot digital assets based on the CoinDesk 5 index. The SEC also approved the listing and trading of PM-settled options on the Cboe Bitcoin US ETF Index and the Mini-Cboe Bitcoin US ETF Index, with expiration dates including third Fridays, non-standard expiration dates, and quarterly index expiration dates.
Paylaş
PANews2025/09/18 07:18
Son of filmmaker Rob Reiner charged with homicide for death of his parents

Son of filmmaker Rob Reiner charged with homicide for death of his parents

FILE PHOTO: Rob Reiner, director of "The Princess Bride," arrives for a special 25th anniversary viewing of the film during the New York Film Festival in New York
Paylaş
Rappler2025/12/16 09:59
3 Shiba Inu Alternatives Crypto Millionaires Are Silently Accumulating in 2025

3 Shiba Inu Alternatives Crypto Millionaires Are Silently Accumulating in 2025

The post 3 Shiba Inu Alternatives Crypto Millionaires Are Silently Accumulating in 2025 appeared on BitcoinEthereumNews.com. Despite its meteoric rise in 2021, Shiba Inu (SHIB) has matured into a large‑cap meme coin with limited room for outsized returns. According to market data, SHIB traded around $0.00001293 on September 20 , 2025, and had a market capitalization of roughly $7.62 billion. With over 589 trillion tokens in circulation and trading volumes in the hundreds of millions, SHIB offers stability but lacks the explosive upside that early adopters crave. As a result, crypto millionaires are quietly rotating capital into smaller, high‑potential projects. Three of the most widely accumulated alternatives are Little Pepe (LILPEPE), Bonk (BONK), and Sui (SUI)—tokens that pair innovative technology or strong community dynamics with significantly lower valuations. Little Pepe (LILPEPE): A presale‑backed memecoin with real infrastructure Little Pepe made headlines in September 2025 when it completed the twelfth stage of its presale, having raised over $25.48 million and distributed more than 15.75 billion tokens. The project immediately moved to stage 13 at a token price of $0.0022, marking a 120 percent increase from the first presale stage. Participants expect further upside because the confirmed listing price is $0.003, implying a 30% gain for Stage-13 buyers. Little Pepe isn’t just another meme coin; it operates on a purpose-built Layer 2 network designed to deliver high-speed, low-cost transactions. The project integrates launchpad functionality for new tokens and includes anti-sniper protection to ensure fair trading. A Certik audit and other independent reviews reinforce its security credentials. This mix of infrastructure and meme culture appeal has attracted significant presale investments—an early signal that influential investors expect LILPEPE to outgrow its current small market capitalization. Bonk, launched on Christmas 2022 as a holiday airdrop to the Solana community, has become Solana’s “main dog‑themed memecoin”. It has embedded itself in the Solana DeFi ecosystem and now counts nearly 983,000 holders. Real‑time data show…
Paylaş
BitcoinEthereumNews2025/09/29 05:19