Enviar pesquisa
Carregar
LOFAR - finding transients in the radio spectrum
•
Transferir como ODP, PDF
•
0 gostou
•
560 visualizações
Gijs Molenaar
Seguir
Tecnologia
Denunciar
Compartilhar
Denunciar
Compartilhar
1 de 18
Baixar agora
Recomendados
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Recomendados
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
Mais conteúdo relacionado
Mais procurados
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Mais procurados
(20)
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Gaucheで本を作る
Gaucheで本を作る
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
2010 jan 12
2010 jan 12
The impact of supercomputers on MSR
The impact of supercomputers on MSR
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
C coroutine
C coroutine
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Semelhante a LOFAR - finding transients in the radio spectrum
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Spark Summit
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Server Density
Apache Spark v3.0.0
Apache Spark v3.0.0
Jean-Georges Perrin
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Ivan Kruglov
Graphite
Graphite
Oleg Obleukhov
Riak add presentation
Riak add presentation
Ilya Bogunov
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
MongoDB
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Neville Li
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
GeeksLab Odessa
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Steven Simpson
Scalable IoT platform
Scalable IoT platform
Swapnil Bawaskar
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Xin Wang
Hadoop at aadhaar
Hadoop at aadhaar
Regunath B
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Víctor Zabalza
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax Academy
JavaOne_2010
JavaOne_2010
Tadaya Tsuyukubo
Semelhante a LOFAR - finding transients in the radio spectrum
(20)
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Apache Spark v3.0.0
Apache Spark v3.0.0
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Graphite
Graphite
Riak add presentation
Riak add presentation
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Scalable IoT platform
Scalable IoT platform
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Hadoop at aadhaar
Hadoop at aadhaar
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
JavaOne_2010
JavaOne_2010
Último
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
Mattias Andersson
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
Rizwan Syed
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
Commit University
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
DianaGray10
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
Lorenzo Miniero
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
hariprasad279825
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
NavinnSomaal
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
Fwdays
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
Sergiu Bodiu
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
Lonnie McRorey
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
charlottematthew16
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
Lars Bell
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
comworks
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Mark Simos
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
Mark Billinghurst
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
ScyllaDB
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
Dilum Bandara
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
RankYa
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
Pixlogix Infotech
Último
(20)
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
LOFAR - finding transients in the radio spectrum
1.
LOFAR Transient Detection
Pipeline Gijs Molenaar gijs@pythonic.nl
2.
Agenda ●
Transient detection ● Pipeline layout ● Software
3.
Transients Detection ●
Static sources are boring ● Transient – something that changes ● Huge amount of data – automation
4.
Image data
5.
The data ●
1 datacube per second ● 10 frequency bands ● In the future 10 images per second ● In the future 4 different polarization ● Non stop
6.
Transient detection Pipeline
Extract metadata Quality check Source extraction Source extraction Association Detection Classification Database
7.
●
The glue ● Distributing computation ● Image processing ● Statistics ● Source extraction ● Database interactions
8.
Distributed Computation ●
Home made libary ● SSH based ● Difficult to debug ● Difficult to profile ● Doing research on Celery and Hadoop
9.
Database ●
Move calculation to the data ● Highly structured ● Independent data ● Naturally separable by sky coordinates ● ~100 TB/year ● 10.000 insert/second
10.
MonetDB ●
Relational ● Column store DB ● Fast ● Auto tuning! ● Developers next door (CWI)
11.
Challenges ●
Debugging queries ● MonetDB still in active development
12.
INSERT INTO tempbasesources (xtrsrc_id ,datapoints ,I_peak_sum ,I_peak_sq_sum ,weight_peak_sum ,weight_I_peak_sum ,weight_I_peak_sq_sum ) SELECT
b0.xtrsrc_id ,b0.datapoints + 1 AS datapoints ,b0.I_peak_sum + x0.I_peak AS i_peak_sum ,b0.I_peak_sq_sum + x0.I_peak * x0.I_peak AS i_peak_sq_sum ,b0.weight_peak_sum + 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum ,b0.weight_I_peak_sum + x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sum ,b0.weight_I_peak_sq_sum + x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sq_sum FROM basesources b0 ,extractedsources x0 WHERE x0.image_id = @imageid AND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight ) AS INTEGER) AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight ) AS INTEGER) AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x) +(x0.y - b0.y)*(x0.y - b0.y) +(x0.z - b0.z)*(x0.z - b0.z) )/2 ) / SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err +x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err) < @assoc_r;
13.
MonetDB and Python ●
We maintain the MonetDB Python API ● http://pypi.python.org/pypi/python-monetdb/ ● Problems? Ask me :)
14.
Djonet ●
MonetDB backend for Django ● https://github.com/gijzelaerr/djonet ● brew install monetdb ● pip install python-monetdb djonet ● Contributions are welcome!
15.
VO events ●
Standardized language ● Report observations of astronomical events ● Hey world, check this supernova out over there ● http://comet.transientskp.org
16.
Visualisation ●
Web interface ● Django! ● Not public (yet)
17.
More ●
http://www.transientskp.org/ ● http://www.lofar.org/ ● http://www.aartfaac.org/
18.
Questions?
Baixar agora