Schakko retweeted
I work on profilers. This is an immediate bookmark for me. This is a really great summary. And a reminder what a mistake DWARF unwinding. Omitting frame pointers on x86-64 was a mistake!
I'm embarrassed to say it, but I didn't have a robust intuition about stack frames, unwinding and exception handling stuff (CIE, FDE, CFI, eh_frame and especially all things DWARF). This great note made it much clearer:
lesenechal.fr/en/linux/unwin…
Schakko retweeted
I've been reverse engineering the xz backdoor this weekend and have documented the payload format and written a proof-of-concept exploit for the RCE. The payloads are signed with an ED448 key, so I patched my own key into the backdoor for testing. :-)
github.com/amlweems/xzbot
Schakko retweeted
This week was incredible for AI speed breakthroughs!
Preparing for @thursdai_pod later today, I combined super fast LLM (<300ms @GroqInc ) + super fast SDXL (thanks to friends at @fal_ai_data) with infinite craft by @nealagarwal to get near instant visualizations of the concepts!
Mixtral via @GroqInc is being used at my prompt engineer, combining the concepts.
SDXL via @fal_ai_data generates the imagery
And I can't stop playing with this!
if someone asked "why do you even need 400 tokens/sec" here's my first attempt at an answer.
Going to cover this on #thursdAI today of course, potentially with folks from Groq AND Fal 👏
Schakko retweeted
🤔 Have you heard about CI/CD observability?
If you're new to it, learn why it's so important and find out what we're working on to help teams everywhere observe their CI/CD pipelines better. bit.ly/3upet0i
Schakko retweeted
We remain committed to our partnership with OpenAI and have confidence in our product roadmap, our ability to continue to innovate with everything we announced at Microsoft Ignite, and in continuing to support our customers and partners. We look forward to getting to know Emmett Shear and OAI's new leadership team and working with them. And we’re extremely excited to share the news that Sam Altman and Greg Brockman, together with colleagues, will be joining Microsoft to lead a new advanced AI research team. We look forward to moving quickly to provide them with the resources needed for their success.
Schakko retweeted
Not your typical SQLi vector... 😈
I've finally published the advisories regarding the Trend Micro bugs that I shared at #HITCON! Do check them out at @starlabs_sg's advisory page: starlabs.sg/advisories/
🏌️♂️CVE-2023-32530 is an interesting case of SQLi to RCE: starlabs.sg/advisories/23/23…
Schakko retweeted
#PostgresMarathon day 15: How to monitor CREATE INDEX / REINDEX progress in Postgres 12+
// I post a new @PostgreSQL "howto" article every day. Join me in this journey – subscribe, provide feedback, share!
To monitor the progress of long-running index building or rebuilding, you can use this query:
select
now(),
query_start as started_at,
now() - query_start as query_duration,
format('[%s] %s', a.pid, a.query) as pid_and_query,
index_relid::regclass as index_name,
relid::regclass as table_name,
(pg_size_pretty(pg_relation_size(relid))) as table_size,
nullif(wait_event_type, '') || ': ' || wait_event as wait_type_and_event,
phase,
format(
'%s (%s of %s)',
coalesce((round(100 * blocks_done::numeric / nullif(blocks_total, 0), 2))::text || '%', 'N/A'),
coalesce(blocks_done::text, '?'),
coalesce(blocks_total::text, '?')
) as blocks_progress,
format(
'%s (%s of %s)',
coalesce((round(100 * tuples_done::numeric / nullif(tuples_total, 0), 2))::text || '%', 'N/A'),
coalesce(tuples_done::text, '?'),
coalesce(tuples_total::text, '?')
) as tuples_progress,
current_locker_pid,
(select nullif(left(query, 150), '') || '...' from pg_stat_activity a where a.pid = current_locker_pid) as current_locker_query,
format(
'%s (%s of %s)',
coalesce((round(100 * lockers_done::numeric / nullif(lockers_total, 0), 2))::text || '%', 'N/A'),
coalesce(lockers_done::text, '?'),
coalesce(lockers_total::text, '?')
) as lockers_progress,
format(
'%s (%s of %s)',
coalesce((round(100 * partitions_done::numeric / nullif(partitions_total, 0), 2))::text || '%', 'N/A'),
coalesce(partitions_done::text, '?'),
coalesce(partitions_total::text, '?')
) as partitions_progress,
(
select
format(
'%s (%s of %s)',
coalesce((round(100 * n_dead_tup::numeric / nullif(reltuples::numeric, 0), 2))::text || '%', 'N/A'),
coalesce(n_dead_tup::text, '?'),
coalesce(reltuples::int8::text, '?')
)
from pg_stat_all_tables t, pg_class tc
where t.relid = p.relid and tc.oid = p.relid
) as table_dead_tuples
from pg_stat_progress_create_index p
left join pg_stat_activity a on a.pid = p.pid
order by p.index_relid
; -- in psql, use "\watch 5" instead of semicolon
The same query, in a better formatted form: gitlab.com/-/snippets/213841…
How this query works:
1. The basis of it is "pg_stat_progress_create_index" added in Postgres 12: postgresql.org/docs/current/…
2. The documentation also has a list of CREATE INDEX phases. As you can see from the table provided in the docs, advanced variant, CREATE INDEX CONCURRENTLY / REINDEX CONCURRENTLY (a.k.a. CIC and RC), which takes longer but acts in a non-blocking fashion suitable for loaded production systems, has more phases. The current phase is presented in the column "phase" of the output.
3. Index name (a temporary one in case of CIC/RC), table name are presented (using the useful trick to convert OIDs to names – note, e.g., "index_relid::regclass as index_name"). Additionally, the table size which is essential to form expectations of overall duration – the bigger the table is, the longer the index creation is going to take.
4. pg_stat_activity (pgsa) provides a lot of additional useful information:
- PID of Postgres backend
- actual SQL query used
- the moment when the work has started (query_start), allowing us to understand the elapsed time (query_duration)
- wait_event_type & wait_event to understand what the process is currently waiting on
- it also used (in a separate sub-query) to get the information of the session that blocks our process, when such an event occurs (current_locker_pid, current_locker_query)
5. Function format(...) is very useful to consolidate data in convenient form without having to worry about NULLs, that would be a problem if we used a regular concatenation without coalesce(...).
6. However, in certain cases we use coalesce(...) just to put special symbols if the value is missing (IS NULL) – e.g., "?" or "N/A".
7. Another interesting trick is combination of coalesce(..) and nullif(..). The latter allows us to avoid division by zero errors (replacing 0 with NULL, making the division result also NULL), while the former, again, is used to substitute resulting NULL with some non-empty value (in this case, 'N/A').
When executed in psql, it can be convenient to use \watch [seconds] to run this report in loop and observe progress live:
Schakko retweeted
SPOILER ALERT 🚨
Microsoft Postgres team is set to release the ai_embedding extension soon, which will complement pgvector, making Postgres an even more attractive option for AI use cases.
So, what exactly does this extension aim to address?
Imagine you store Airbnb lodging options in Postgres, including embeddings/vectors for each lodging description. When a user searches for a place to stay, presently, you can use OpenAI + Postgres for natural language processing in the manner below.
First, generate an embedding for the user's prompt:
const embeddingResp = await this.#openai.embeddings.create(
{ model: "text-embedding-ada-002",
input: "Looking for an apartment in NYC"
}
);
Next, take the generated embedding and do a similarity search in Postgres to provide the most relevant recommendations for the user:
const res = await this.#client.query(
"SELECT name, description, price " +
"FROM airbnb_listing WHERE 1 - (description_embedding <=> $1) > 0.8 " +
"ORDER BY similarity DESC LIMIT 3",
['[' + userPromptEmbedding + ']']);
I've detailed this process in this article: medium.com/@magda7817/better…
Now, what changes with the upcoming Microsoft extension? You'll be able to call the OpenAI API directly from the database, straight from SQL, making the code more streamlined and efficient.
const res = await this.#client.query(
"SELECT name, description, price " +
"FROM airbnb_listing WHERE 1 - " +
"(description_embedding <=> " +
"ai_embedding('Looking for an apartment in NYC') > 0.8 " +
"ORDER BY similarity DESC LIMIT 3”);
Good luck to your team @clairegiordano . Thanks for making Postgres awesome.
Schakko retweeted
Introducing #Kargo - a multi-stage application lifecycle orchestrator for continuously delivering and promoting changes through environments. Find out more from a post by @JesseSuen and join tomorrow's webinar for a live demo with @kelseyhightower 🥳
hubs.li/Q022wjnR0
Schakko retweeted
🛡️ Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines
37 page PDF from @NIST
* Risk factors and mitigation measures
* CI/CD security goals
* Securing workflows in CI pipelines
+ more
csrc.nist.gov/pubs/sp/800/20…
Schakko retweeted
The Otamatone is an electronic musical toy developed in Japan known for its unique design resembling an eighth note, accompanied by an endearing animated mouth
Listen to the sound of creativity
📹@moshibass
Schakko retweeted
Explore the potential of the Object Capture feature with iOS 17. Our second demo video witnesses the transformation of a drinking fountain in Paris into a stunning 3D model, all directly from your iPhone or iPad Pro. Our upcoming AR Code app update promises to further streamline this process, making it simpler than ever to create AR Codes from your scanned objects. Stay tuned for its release at ar-code.com #iOS17 #ARCode #3DScanning #Photogrammetry #AugmentedReality #Paris
Schakko retweeted
The @OCI_ORG Specification, basically.
Schakko retweeted
Wir trauern um Stefan Tilkov.
We mourn the loss of Stefan Tilkov.
Deutsch: innoq.com/de/news/2023/08/st…
English: innoq.com/en/news/2023/08/st…
































