A5074160158- Distributed and Parallel Computing Systems
- Advanced Data Storage Technologies
- Scientific Computing and Data Management
- Particle physics theoretical and experimental studies
- Particle Detector Development and Performance
- Multi-Agent Systems and Negotiation
- Mobile Agent-Based Network Management
Instituto de Física Corpuscular
2016-2025
Universitat de València
2017-2025
Institute for Cross-Disciplinary Physics and Complex Systems
2019
The ATLAS Eventlndex is the global catalogue of all real and simulated events. During LHC long shutdown between Run 2 (20152018) 3 (2022-2025) its components were substantially revised a new system was deployed for start in Spring 2022. core storage system, based on HBase tables with SQL interface provided by Phoenix, allows much faster data ingestion rates scales better than old one to expected end beyond. All user interfaces also command-line web services deployed. initially populated...
The ATLAS EventIndex system comprises the catalogue of all events collected, processed or generated by experiment at CERN LHC accelerator, and associated software tools to collect, store query this information. records several billion particle interactions every year operation, processes them for analysis generates even larger simulated data samples; a global is needed keep track location each event record be able search retrieve specific in-depth investigations. Each includes summary...
The ATLAS EventIndex is the global catalogue of all real and simulated events. During LHC long shutdown between Run 2 (20152018) 3 (2022-2025) its components were substantially revised, a new system was deployed for start in Spring 2022. core storage based on HBase tables with Phoenix interface. It allows faster data ingestion rates scales better than old system. This paper describes collection, technical design storage, properties that make it fast efficient, namely compact optimized events...
The ATLAS EventIndex currently runs in production order to build a complete catalogue of events for experiments with large amounts data. current approach is index all final produced data files at CERN Tier0, and hundreds grid sites, distributed collection architecture using Object Stores temporarily maintain the conveyed information, references them sent Messaging System. backend indexed central Hadoop infrastructure CERN; an Oracle relational database used faster access subset this...
The ATLAS EventIndex has been in operation since the beginning of LHC Run 2 2015. Like all software projects, its components have constantly evolving and improving performance. main data store Hadoop, based on MapFiles HBase, can work for rest but new solutions are explored future. Kudu offers an interesting environment, with a mixture BigData relational database features, which look promising at design level. This environment is used to build prototype measure scaling capabilities as...
The ATLAS EventIndex is a data catalogue system that stores event-related metadata for all (real and simulated) events, on processing stages. As it consists of different components depend other applications (such as distributed storage, sources information) we need to monitor the conditions many heterogeneous subsystems, make sure everything working correctly. This paper describes how gather information about related subsystems: Producer-Consumer architecture collection, health parameters...
The ATLAS EventIndex has been running in production since mid-2015, reliably collecting information worldwide about all produced events and storing them a central Hadoop infrastructure at CERN. A subset of this is copied to an Oracle relational database for fast dataset discovery, event-picking, crosschecks with other systems checks event duplication. system design its optimization serving picking from requests few up scales tens thousand events, addition, data consistency are performed...
The ATLAS Spanish Tier-1 and Tier-2s have more than 18 years of experience in the deployment development LHC computing components their successful operation. sites are actively participating in, some cases coordinating, R&D activities Run 3 developing models needed HL-LHC period. In this contribution, we present details on integration components, such as HPC resources to execute simulation workflows; new techniques improve efficiency a cost-effective way; improvements Data Organization,...
The ATLAS EventIndex was designed in 2012-2013 to provide a global event catalogue and limited event-level metadata for analysis groups users during the LHC Run 2 (2015-2018). It provides good reliable service initial use cases (mainly picking) several additional ones, such as production consistency checks, duplicate detection measurements of overlaps trigger chains derivation datasets. 3, starting 2021, will see increased data-taking simulation rates, with which current infrastructure would...
The ATLAS EventIndex was designed to provide a global event catalogue and limited event-level metadata for experiment of the Large Hadron Collider (LHC) their analysis groups users during Run 2 (2015-2018) has been running in production since. LHC 3, started 2022, seen increased data-taking simulation rates, with which current infrastructure would still cope but may be stretched its limits by end 3. A new core storage service is being developed HBase/Phoenix, there work progress at least...
The ATLAS EventIndex provides a global event catalogue and event-level metadata for analysis groups users. LHC Run 3, starting in 2022, will see increased data-taking simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by end of 3. This talk describes implementation new core storage service that provide at least same functionality as one data ingestion search increasing volumes stored data. It is based on set HBase tables,...
Since the beginning of WLCG Project Spanish ATLAS computing centers have participated with reliable and stable resources as well personnel for Collaboration. Our contribution to Tier2s Tier1s (disk CPUs) in last 10 years has been around 4-5%. In 2016 an international advisory committee recommended revise our according participation experiment. With this scenario, we are optimizing federation three sites located Barcelona, Madrid Valencia, considering that collaboration developed workflows...
Abstract The ATLAS experiment has produced hundreds of petabytes data and expects to have one order magnitude more in the future. This are spread among computing Grid sites around world. EventIndex is complete catalogue all events, real simulated, keeping references permanent files that contain a given event any processing stage. It provides means select access distributed storage system, support for completeness consistency checks trigger offline selection overlap studies. employs various...
The ATLAS Spanish Tier-1 and Tier-2s have more than 15 years of experience in the deployment development LHC computing components their successful operations. sites are already actively participating in, even coordinating, emerging R&D activities developing new models needed for Run3 HighLuminosity periods. In this contribution, we present details on integration components, such as High Performance Computing resources to execute simulation workflows. techniques improve efficiency a...
The ATLAS EventIndex is the catalogue of event-related metadata for information collected from detector. basic unit this event record, containing identification parameters, pointers to files as well trigger decision information. main use case picking, data consistency checks large production campaigns. employs Hadoop platform storage and handling, a messaging system collection both at Tier-0, when are first produced, Grid, various types derived produced. uses auxiliary other sources...
The Event Index service of the ATLAS experiment at LHC keeps references to all real and simulated events. Hadoop Map files HBase tables are used store data, a subset data is also stored in Oracle database. Several user interfaces currently access search from simple command line interface, through programmable API, sophisticated graphical web services. It provides dynamic graph-like overview available (and collections). Data shown together with their relations, like paternity or overlaps....