
#NLP #Multimodal #AIScientists #AgentSecurity Scientist @PNNLab PhD @cseUSF Opinions here are my own and do not represent my employer. Proud 🇱🇰 Live 🇺🇸
Tampa, FL
Joined November 2010
- Tweets5.4K
- Following904
- Followers711
- Likes3.7K
Sameera Horawalavithana retweeted
The next frontier is AI that improves itself, and we believe everyone should know how to build it.
This semester, Ao @ao_qu18465 and I are teaching 'How to AI an AI for Almost Anything' @MIT @medialab -- a new course on self-evolving AI systems that recursively improve from experience, feedback, memory, and exploration over long interaction horizons.
The key idea: intelligence shouldn't stop improving when training ends.
We cover four levels of evolution:
(1) context evolution through prompt optimization and context management,
(2) harness evolution through memory, tools & skills, workflow optimization, and multi-agent systems,
(3) model evolution through RL, distillation, test-time training, self-play, and continual learning,
(4) task evolution through generating increasingly complex synthetic tasks, simulations, evaluations, and environments.
Slides, readings, and videos available: mit-mi.github.io/how2ai-cour…
We thank @Kimi_Moonshot and @PrimeIntellect for generously sponsoring API credits!
PhD internships opportunities to work with the language intelligence team @PNNLab to work on AI for science and legal/permitting domains. please apply. #Internship #NLP
careers.pnnl.gov/jobs/12149?…
🧵 1/ New preprint drop: "Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models" 🦙
If you swap a better LLM backbone into your VLM, do you get a better VLM?
Short answer: not really. Longer answer: it's more interesting than that.
1
1
114
4/ some VLM capabilities only emerge in the newest LLM generation. Other tasks dominated by visual understanding barely improve regardless of which LLAMA you plug in
1
62
Read More. arxiv.org/abs/2604.10985
Sameera Horawalavithana retweeted
Streamlining federal permitting with AI 📄🖥️⏩
PNNL researchers are using AI to bring valuable data distributed across hundreds of federal government agencies into a single dataset that's crucial for modernizing permitting technology for the 21st century.
Sameera Horawalavithana retweeted
CEQ coordinated with @ENERGY’s @PNNLab PermitAI project on the release of NEPATEC 2.0, a major accomplishment on the road to a simplified, speedier Federal permitting and environmental review process.
We are organizing AI4Permitting Workshop on April 29th at the NAEP 2025 conference in Charleston, South Carolina.
Learn how to speed up National Environmental Policy Act (NEPA) processes at our upcoming workshop at NAEP 2025.
pnnl.gov/events/using-nepa-c…
#AI4Permitting #NEPA
86
Sameera Horawalavithana retweeted
We also currently have a kaggle competition open, which is running LLM evaluations on a QuAD benchmark specific to understanding permitting documents - deadline is june 30!
kaggle.com/competitions/llm-…
Sameera Horawalavithana retweeted
This corpus is part of DOE's voltAIc initiative, which is using LLMs to accelerate permitting processes
This corpus was developed in partnership with @PNNLab - you can find more details about their work on this project here:
pnnl.gov/projects/policyai
Sameera Horawalavithana retweeted
🚨if you care about fixing NEPA environmental permitting OR are looking for new high-quality domain text corpuses to build LLMs on top of...
DOE just released a 3.6B token corpus of federal permitting documents on huggingface!
This corpus includes...
Sameera Horawalavithana retweeted
The video of the Lytle Lecture I gave at University of Washington last week is available.
Title: "Objective Driven AI: Towards Machines that can Learn, Reason, and Plan"
Lytle Lecture Page: ece.uw.edu/news-events/lytle…
Slides: drive.google.com/file/d/1e6E…
Video: youtu.be/d_bdU3LsLzE?si=UeLf…
Sameera Horawalavithana retweeted
📢 Accepted papers are up!
🔗: an-instructive-workshop.gith…
Check out the range of cool new work on instruction training!
Sameera Horawalavithana retweeted
📢📢 I am recruiting PhD students for our group at Auburn to work on open-world event understanding and active embodied vision! If you are interested in working with our group, reach out!
More info: saakur.github.io/group.html
Please RT!
Sameera Horawalavithana retweeted
A new multimodal model LLaMA-SciTune for science-focused visual and language understanding.
📄 arxiv.org/abs/2307.01139v1
Sameera Horawalavithana retweeted
SCITUNE: Aligning Large Language Models with Scientific Multimodal
Instructions
Abs: arxiv.org/abs/2307.01139
Pdf: arxiv.org/pdf/2307.01139.pdf
Presenting SciTune, a tuning framework to improve large language models' (LLMs) ability to follow scientific multimodal instructions. SciTune includes two stages: scientific concept alignment to learn across various scientific visual signals and textual signals, and scientific instruction tuning to fine-tune on a multimodal scientific reasoning task. LLaMA-SciTune, surpasses human performance on the ScienceQA multimodal reasoning benchmark and performs significantly better than SoTA vision-language models in a variety of scientific image understanding tasks with zero-demonstrations during the inference time.
Sameera Horawalavithana retweeted
Almost everyone I know working in AI these days feels one step away from total burnout. I took the time to take you behind the curtain and know what people on the state-of-the-art AI are struggling with:
robotic.substack.com/p/behin…
Sameera Horawalavithana retweeted
Really nice survey on augmenting language models with reasoning skills and the ability to use tools.
Includes prompting techniques, use cases, and applications.
A must-read!
arxiv.org/abs/2302.07842


















