Word of the week: Big Data
This week our chief Architect, Tim Cornwell, tells us a bit more about the IT end of the SKA telescope and its challenges by explaining what we mean when we talk about “Big Data”:
“In a normal scientific instrument, most of the effort goes into building the instrument itself, and the data gathered can be analysed using common tools, including even spreadsheet programs.
A big data instrument like the SKA has different challenges. Composed of many antennas, the SKA will gather a lot of data, therefore a lot of effort must go not only in building the telescope itself, but also in dealing with the data it will produce, which will have to be processed in parallel on the equivalent of millions of computers.
Because there will be so much data coming in all the time, and we only have a limited amount of space available, it will also be necessary to process and store the useful data and then delete the rest quickly so that more data can be collected.
Finally, scientists will have to write very sophisticated computer programs to be able to search for what they want in the stored data, like a needle in a very big haystack.
As an example, the SKA’s first phase of development will see the equivalent of 3 TB (about the size of a standard desktop computer disk) transmitted every second to the central processing computer. So that’s Big Data”
See the definition on Facebook.
— SquareKilometreArray (@SKA_telescope) March 24, 2014