Neural sequence labeling is widely adopted for many Natural Language Processing (NLP) tasks, such as Named Entity Recognition (NER) and slot tagging dialog systems semantic parsing. Recent advances with large-scale pre-trained language models have shown remarkable success in these tasks when fine-tuned on large amounts of task-specific labeled data. However, obtaining training data not only costly, but also may be feasible sensitive user applications due to access privacy constraints. This...

Federated Learning has shown great potentials for the distributed data utilization and privacy protection. Most existing federated learning approaches focus on supervised setting, which means all stored in each client labels. However, real-world applications, are impossible to be fully labeled. Thus, how exploit unlabeled should a new challenge learning. Although few studies attempting overcome this challenge, they may suffer from information leakage or misleading usage problems. To tackle...

The broad adoption of electronic health record (EHR) systems and the advances deep learning technology have motivated development risk prediction models, which mainly depend on expressiveness temporal modeling capacity neural networks (DNNs) to improve performance. Some further augment by using external knowledge, however, a great deal EHR information inevitably loses during knowledge mapping. In addition, made existing models usually lacks reliable interpretation, undermines their...

Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds millions to billions parameters, and storing a copy the PLM weights every task resulting in increased cost storing, sharing serving models. To address this, parameter-efficient (PEFT) techniques were introduced where small trainable components are injected updated during fine-tuning. We propose AdaMix as general PEFT method that tunes mixture adaptation modules -- given underlying...

Synthesizing electronic health records (EHR) data has become a preferred strategy to address scarcity, improve quality, and model fairness in healthcare. However, existing approaches for EHR generation predominantly rely on state-of-the-art generative techniques like adversarial networks, variational autoencoders, language models. These methods typically replicate input visits, resulting inadequate modeling of temporal dependencies between visits overlooking the time information, crucial...

Fake news travels at unprecedented speeds, reaches global audiences and puts users communities great risk via social media platforms. Deep learning based models show good performance when trained on large amounts of labeled data events interest, whereas the tends to degrade other due domain shift. Therefore, significant challenges are posed for existing detection approaches detect fake emergent events, where large-scale datasets difficult obtain. Moreover, adding knowledge from newly...

Personalized text generation is an emerging research area that has attracted much attention in recent years. Most studies this direction focus on a particular domain by designing bespoke features or models. In work, we propose general approach for personalized using large language models (LLMs). Inspired the practice of writing education, develop multistage and multitask framework to teach LLMs generation. instruction, task from sources often decomposed into multiple steps involve finding,...

Meta-learning enables learning systems to adapt quickly new tasks, similar humans. To emulate this human-like rapid and enhance alignment discrimination abilities, we propose ConML, a universal meta-learning framework that can be applied various algorithms without relying on specific model architectures nor target models. The core of ConML is task-level contrastive learning, which extends from the representation space in unsupervised meta-learning. By leveraging task identity as an...

In-context learning (ICL) enables large language models (LLMs) to generalize new tasks by incorporating a few in-context examples (ICEs) directly in the input, without updating parameters. However, effectiveness of ICL heavily relies on selection ICEs, and conventional text-based embedding methods are often inadequate for that require multi-step reasoning, such as mathematical logical problem solving. This is due bias introduced shallow semantic similarities fail capture deeper reasoning...

Cross-lingual natural language understanding~(NLU) aims to train NLU models on a source and apply the tasks in target languages, is fundamental task for many cross-language applications. Most of existing cross-lingual assume existence parallel corpora so that words sentences languages could be aligned. However, construction such expensive sometimes infeasible. Motivated by this challenge, recent works propose data augmentation or adversarial training methods reduce reliance external corpora....

Cross-lingual natural language understanding(NLU) is one of the fundamental tasks NLP. The goal to learn a model which can generalize well on both high-resource and low-resource data. Recent pre-trained multilingual models, e.g., BERT, XLM, have shown impressive performance cross-lingual NLU tasks. However, such promising results request use sufficient training data, difficult condition satisfy for language. When data limited in those low resource languages, accuracy existing models will...

The knowledge concept prerequisites describing the dependencies are critical for fundamental tasks such as material recommendations and there a huge amount of concepts in Massive Open Online Courses (MOOCs). Thus it is necessary to develop automatic prerequisite relation annotation methods. Recently, few methods have shown their effectiveness discovering Moocs automatically. However, they suffer from two common issues, i.e., not thoroughly learnt, informative supervision sources ignored. To...

In this work, we study the problem of named entity recognition (NER) in a low resource scenario, focusing on few-shot and zero-shot settings. Built upon large-scale pre-trained language models, propose novel NER framework, namely SpanNER, which learns from natural supervision enables identification never-seen classes without using in-domain labeled data. We perform extensive experiments 5 benchmark datasets evaluate proposed method learning, domain transfer learning The experimental results...