Enviar pesquisa
Carregar
LOFAR - finding transients in the radio spectrum
•
Transferir como ODP, PDF
•
0 gostou
•
560 visualizações
Gijs Molenaar
Seguir
Tecnologia
Denunciar
Compartilhar
Denunciar
Compartilhar
1 de 18
Baixar agora
Recomendados
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Recomendados
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
Mais conteúdo relacionado
Mais procurados
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Mais procurados
(20)
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Gaucheで本を作る
Gaucheで本を作る
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
2010 jan 12
2010 jan 12
The impact of supercomputers on MSR
The impact of supercomputers on MSR
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
C coroutine
C coroutine
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Semelhante a LOFAR - finding transients in the radio spectrum
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Spark Summit
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Server Density
Apache Spark v3.0.0
Apache Spark v3.0.0
Jean-Georges Perrin
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Ivan Kruglov
Graphite
Graphite
Oleg Obleukhov
Riak add presentation
Riak add presentation
Ilya Bogunov
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
MongoDB
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Neville Li
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
GeeksLab Odessa
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Steven Simpson
Scalable IoT platform
Scalable IoT platform
Swapnil Bawaskar
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Xin Wang
Hadoop at aadhaar
Hadoop at aadhaar
Regunath B
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Víctor Zabalza
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax Academy
JavaOne_2010
JavaOne_2010
Tadaya Tsuyukubo
Semelhante a LOFAR - finding transients in the radio spectrum
(20)
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Apache Spark v3.0.0
Apache Spark v3.0.0
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Graphite
Graphite
Riak add presentation
Riak add presentation
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Scalable IoT platform
Scalable IoT platform
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Hadoop at aadhaar
Hadoop at aadhaar
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
JavaOne_2010
JavaOne_2010
Último
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
Remote DBA Services
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Martijn de Jong
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
Michael W. Hawkins
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
Maria Levchenko
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
hans926745
Evaluating the top large language models.pdf
Evaluating the top large language models.pdf
ChristopherTHyatt
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Enterprise Knowledge
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
The Digital Insurer
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Igalia
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Anna Loughnan Colquhoun
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
Antenna Manufacturer Coco
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Delhi Call girls
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Principled Technologies
Último
(20)
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
Evaluating the top large language models.pdf
Evaluating the top large language models.pdf
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
LOFAR - finding transients in the radio spectrum
1.
LOFAR Transient Detection
Pipeline Gijs Molenaar gijs@pythonic.nl
2.
Agenda ●
Transient detection ● Pipeline layout ● Software
3.
Transients Detection ●
Static sources are boring ● Transient – something that changes ● Huge amount of data – automation
4.
Image data
5.
The data ●
1 datacube per second ● 10 frequency bands ● In the future 10 images per second ● In the future 4 different polarization ● Non stop
6.
Transient detection Pipeline
Extract metadata Quality check Source extraction Source extraction Association Detection Classification Database
7.
●
The glue ● Distributing computation ● Image processing ● Statistics ● Source extraction ● Database interactions
8.
Distributed Computation ●
Home made libary ● SSH based ● Difficult to debug ● Difficult to profile ● Doing research on Celery and Hadoop
9.
Database ●
Move calculation to the data ● Highly structured ● Independent data ● Naturally separable by sky coordinates ● ~100 TB/year ● 10.000 insert/second
10.
MonetDB ●
Relational ● Column store DB ● Fast ● Auto tuning! ● Developers next door (CWI)
11.
Challenges ●
Debugging queries ● MonetDB still in active development
12.
INSERT INTO tempbasesources (xtrsrc_id ,datapoints ,I_peak_sum ,I_peak_sq_sum ,weight_peak_sum ,weight_I_peak_sum ,weight_I_peak_sq_sum ) SELECT
b0.xtrsrc_id ,b0.datapoints + 1 AS datapoints ,b0.I_peak_sum + x0.I_peak AS i_peak_sum ,b0.I_peak_sq_sum + x0.I_peak * x0.I_peak AS i_peak_sq_sum ,b0.weight_peak_sum + 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum ,b0.weight_I_peak_sum + x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sum ,b0.weight_I_peak_sq_sum + x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sq_sum FROM basesources b0 ,extractedsources x0 WHERE x0.image_id = @imageid AND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight ) AS INTEGER) AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight ) AS INTEGER) AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x) +(x0.y - b0.y)*(x0.y - b0.y) +(x0.z - b0.z)*(x0.z - b0.z) )/2 ) / SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err +x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err) < @assoc_r;
13.
MonetDB and Python ●
We maintain the MonetDB Python API ● http://pypi.python.org/pypi/python-monetdb/ ● Problems? Ask me :)
14.
Djonet ●
MonetDB backend for Django ● https://github.com/gijzelaerr/djonet ● brew install monetdb ● pip install python-monetdb djonet ● Contributions are welcome!
15.
VO events ●
Standardized language ● Report observations of astronomical events ● Hey world, check this supernova out over there ● http://comet.transientskp.org
16.
Visualisation ●
Web interface ● Django! ● Not public (yet)
17.
More ●
http://www.transientskp.org/ ● http://www.lofar.org/ ● http://www.aartfaac.org/
18.
Questions?
Baixar agora