
#NLP #Multimodal #AIScientists #AgentSecurity Scientist @PNNLab PhD @cseUSF Opinions here are my own and do not represent my employer. Proud π±π° Live πΊπΈ
Tampa, FL
Joined November 2010
- Tweets5.4K
- Following904
- Followers711
- Likes3.7K
Sameera Horawalavithana retweeted
The next frontier is AI that improves itself, and we believe everyone should know how to build it.
This semester, Ao @ao_qu18465 and I are teaching 'How to AI an AI for Almost Anything' @MIT @medialab -- a new course on self-evolving AI systems that recursively improve from experience, feedback, memory, and exploration over long interaction horizons.
The key idea: intelligence shouldn't stop improving when training ends.
We cover four levels of evolution:
(1) context evolution through prompt optimization and context management,
(2) harness evolution through memory, tools & skills, workflow optimization, and multi-agent systems,
(3) model evolution through RL, distillation, test-time training, self-play, and continual learning,
(4) task evolution through generating increasingly complex synthetic tasks, simulations, evaluations, and environments.
Slides, readings, and videos available: mit-mi.github.io/how2ai-courβ¦
We thank @Kimi_Moonshot and @PrimeIntellect for generously sponsoring API credits!
Sameera Horawalavithana retweeted
Streamlining federal permitting with AI ππ₯οΈβ©
PNNL researchers are using AI to bring valuable data distributed across hundreds of federal government agencies into a single dataset that's crucial for modernizing permitting technology for the 21st century.
Sameera Horawalavithana retweeted
CEQ coordinated with @ENERGYβs @PNNLab PermitAI project on the release of NEPATEC 2.0, a major accomplishment on the road to a simplified, speedier Federal permitting and environmental review process.
Sameera Horawalavithana retweeted
We also currently have a kaggle competition open, which is running LLM evaluations on a QuAD benchmark specific to understanding permitting documents - deadline is june 30!
kaggle.com/competitions/llm-β¦
Sameera Horawalavithana retweeted
This corpus is part of DOE's voltAIc initiative, which is using LLMs to accelerate permitting processes
This corpus was developed in partnership with @PNNLab - you can find more details about their work on this project here:
pnnl.gov/projects/policyai
Sameera Horawalavithana retweeted
π¨if you care about fixing NEPA environmental permitting OR are looking for new high-quality domain text corpuses to build LLMs on top of...
DOE just released a 3.6B token corpus of federal permitting documents on huggingface!
This corpus includes...
Sameera Horawalavithana retweeted
The video of the Lytle Lecture I gave at University of Washington last week is available.
Title: "Objective Driven AI: Towards Machines that can Learn, Reason, and Plan"
Lytle Lecture Page: ece.uw.edu/news-events/lytleβ¦
Slides: drive.google.com/file/d/1e6Eβ¦
Video: youtu.be/d_bdU3LsLzE?si=UeLfβ¦
Sameera Horawalavithana retweeted
π’ Accepted papers are up!
π: an-instructive-workshop.githβ¦
Check out the range of cool new work on instruction training!
Sameera Horawalavithana retweeted
π’π’ I am recruiting PhD students for our group at Auburn to work on open-world event understanding and active embodied vision! If you are interested in working with our group, reach out!
More info: saakur.github.io/group.html
Please RT!
Sameera Horawalavithana retweeted
A new multimodal model LLaMA-SciTune for science-focused visual and language understanding.
π arxiv.org/abs/2307.01139v1
Sameera Horawalavithana retweeted
SCITUNE: Aligning Large Language Models with Scientific Multimodal
Instructions
Abs: arxiv.org/abs/2307.01139
Pdf: arxiv.org/pdf/2307.01139.pdf
Presenting SciTune, a tuning framework to improve large language models' (LLMs) ability to follow scientific multimodal instructions. SciTune includes two stages: scientific concept alignment to learn across various scientific visual signals and textual signals, and scientific instruction tuning to fine-tune on a multimodal scientific reasoning task. LLaMA-SciTune, surpasses human performance on the ScienceQA multimodal reasoning benchmark and performs significantly better than SoTA vision-language models in a variety of scientific image understanding tasks with zero-demonstrations during the inference time.
Sameera Horawalavithana retweeted
Almost everyone I know working in AI these days feels one step away from total burnout. I took the time to take you behind the curtain and know what people on the state-of-the-art AI are struggling with:
robotic.substack.com/p/behinβ¦
Sameera Horawalavithana retweeted
Really nice survey on augmenting language models with reasoning skills and the ability to use tools.
Includes prompting techniques, use cases, and applications.
A must-read!
arxiv.org/abs/2302.07842
Sameera Horawalavithana retweeted
Generative AI is climbing the *Dimensional Ladder*. I made a figure to show the milestones!
Text =>
1D: MusicLM, VALL-E
2D: Stable Diffusion, DALL-E, MidJourney
3D (or 2+1D): Imagen-video, Phenaki
3D: Magic3D, DreamFusion, Point-E
4D (or 3+1D): Make-A-Video-3D
Whatβs next? π€
Sameera Horawalavithana retweeted
π₯ New blog: Graph ML papers at #ICML2022 in 10 categories: from denoising diffusion generation through transformers, expressive and explainable GNNs to algorithmic reasoning, knowledge graphs, and cool applications! Jointly with @zhu_zhaocheng
mgalkin.medium.com/graph-macβ¦
Sameera Horawalavithana retweeted
Accepted Papers π #ACL2022 Workshop: πΈ Challenges & Perspectives in Creating Large Language Models
Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned openreview.net/forum?id=SLX-β¦
Sameera Horawalavithana retweeted
Here are some intersting posters I found while attending NeurIPS π (π§΅, 0/N)
Sameera Horawalavithana retweeted
Tomorrow at 9am CST, PNNL researchers are chairing this @Supercomputing workshop on irregular applications. Come see @ascgroup, @mminutoli84, Vito Giovanni Castellana, and John Feo at #SC21! https://teapaw.org/t.co/kZRz0B0NAE
Sameera Horawalavithana retweeted
Hi all! Recruiting again for our study with Youtubers who moderate comments on videos. After interviewing many of you about the challenges, we've now built a tool to make moderation easier, and we'd love to get feedback. $25 for 45 min of your time. Please DM if interested!
This quoted post is unavailable.
Sameera Horawalavithana retweeted
If you have less than 3 hours to spare & want to learn (almost) everything about state-of-the-art explainable ML, this thread is for you! Below, I am sharing info about 4 of our recent tutorials on explainability presented at NeurIPS, AAAI, FAccT, and CHIL conferences. [1/n]
























