Transcription

Standard Big Data Architectureand InfrastructureWo ChangDigital Data AdvisorInformation Technology Laboratory (ITL)National Institute of Standards and Technology (NIST)[email protected] 20, 2016NIST Big Data Standards Activities, Wo Chang, May 20, 20161

Agenda Brief Introduction – background, projects, interests Big Data Architecture and Infrastructure – challenges and opportunities Samples of Independent Big Data Activities Collaboration Focus Areas – Join Big Data international standards development Other Topics: Make application domains (Big Data, IoT, CPS, Smart Cities, etc.) availableNIST Big Data Standards Activities, Wo Chang, May 20, 20162

Brief Introduction: Wo ChangMr. Wo Chang, Digital Data Advisor, NIST Information Tech. Lab. (ITL) Responsibilities: Open data and big data interoperability. Mr. Chang currentlythe Convener of the ISO/IEC JTC 1/WG9 Working Group on Big Data, co-chairsthe NIST Big Data Public Working Group, and chairs the ISO/IEC JTC/1 SC 29WG11 (MPEG) Multimedia Preservation AHG. Prior to joining ITL Office, Mr. Chang was manager of the Digital Media Group inITL and his duties included oversees several key projects including digital dataarchival and preservation, management of electronic health records, motionimage quality, cloud computing, and multimedia standards. In the past, Changwas the Deputy Chair for the US INCITS L3.1, chaired several other keyprojects for MPEG, participated with the HL7 and ISO/IEC TC215 for healthinformatics, IETF for the protocols development, and was one of the originalmembers of the W3C's SMIL and developed one of the SMIL referencesoftware.NIST Big Data Standards Activities, Wo Chang, May 20, 20163

Brief Introduction: Projects and Interests NIST Big Data Public Working Group, Co-ChairISO/IEC JTC 1/WG 9 Working Group on Big Data, ConvenerDevelop a reference architecture that is vendor-neutral, technology- andinfrastructure-agnostic to enable any stakeholders to perform analyticsprocessing for their given data sources without worrying about theunderlying computing environment. NIST Ubiquitous Data Interoperability, Lead ArchitectDevelop an interoperable data infrastructure that is scalable to enableautomatic data mashups between heterogeneous datasets from variousdomains without worrying about the data source and structure. Research InterestsScalable graph mining algorithms and visual analytics for massiveaudiovisual content, digital data mashup, cloud computing, contentmetadata description, multimedia synchronization, and Internet protocols.NIST Big Data Standards Activities, Wo Chang, May 20, 20164

Big Data Architecture and Infrastructure – Challenges (Computing Stack)NIST Big Data Standards Activities, Wo Chang, May 20, 20165

Big Data Architecture and Infrastructure – Challenges (Analytics Stack)Source: http://1.bp.blogspot.com/-PKiTQa0mrn4/T mGb6AI3yI/AAAAAAAAA8Q/TtH7xyjQ3FA/s640/analytics tools landscape.bmpNIST Big Data Standards Activities, Wo Chang, May 20, 20166

Big Data Architecture and Infrastructure – Challenges (Data Stack)NIST Big Data Standards Activities, Wo Chang, May 20, 20167

Big Data Architecture and Infrastructure – Challenges (Integration)Source: 012/10/Big-Data-Landscape.jpgNIST Big Data Standards Activities, Wo Chang, May 20, 20168

Big Data Architecture and Infrastructure – Opportunities (Interoperable Ecosystem)DataScientistResource Management/Monitoring, Analytics Libraries, etc.BDRA InterfaceBDRA Ecosystem ComponentsData Sources- Sensors- Simulations- Modeling- Etc.ComputingResourcesSupport InfrastructureData TypesServicesDatabase ServicesDistributed File System ServicesInfrastructure ServicesData Consumers- End users- Repositories- Systems- Etc.AnalyticsResourcesAnalytics ApplicationAnalytics ServicesVisualization & BI ServicesValue-added Content ServicesSecurity and Privacy ServicesNIST Big Data Standards Activities, Wo Chang, May 20, 20169

Samples of Independent Big Data ActivitiesEU Big Data Value Association (BDVA)TF6: Technical: Identifying and refining the technical challenges of theprogramme – eg Data Management(Created Sub-Group 6 to deal with Big Data standardization)NIST Big Data Standards Activities, Wo Chang, May 20, 201610

Samples of Independent Big Data ActivitiesUS National Science Foundation (NSF) Big Data HubsNortheastSouthMidwestWest Education Health Disparities Food-Water-Energy Big Data technology Cities / Regions Coastal Hazards Health Sciences, Life Managing natural resources Discovery Science IndustrialSciences, Bioinformatics, Energy Materials and ManufacturingGenomics Finance Habitat Planning Health Smart Cities andCommunities Digital Agriculture (precisionand hazards Precision medicine Metro data science Data-enabled scientificdiscovery and learningfarming, sustainability, ) Advanced Manufacturing Network Science Transportation Business Analytics Ring 1: Tools and Services Ring 2: Data ScienceNIST Big Data Standards Activities, Wo Chang, May 20, 201611

Samples of Independent Big Data ActivitiesNIST Big Data Public Working Group (June 2013 – now)Goal: Develop a consensus-basedV1 (high-level NBD-RA components andreference architecture that is vendordescriptions) Big Data Interoperabilityneutral, technology and infrastructureFramework, Released September 16, 2015agnostic to enable any stakeholdershttp://bigdatawg.nist.govto perform analytics processing fortheir given data sources withoutNIST SP1500-3:worrying about the underlyingNIST SP1500-1:NIST SP1500-2:Use Cases &computing ities – 5 Subgroups1. Definitions & Taxonomies2. Use Cases & Requirements3. Security & Privacy4. Reference Architecture5. Standards RoadmapNIST SP1500-4:Security &PrivacyNIST SP1500-5:ArchitectureSurvey – WhitePaperNIST SP1500-6:ReferenceArchitectureNIST SP1500-7:StandardsRoadmapNIST Big Data Standards Activities, Wo Chang, May 20, 201612

Samples of Independent Big Data ActivitiesISO/IEC JTC 1/WG 9 Working Group on Big DataSummary Key Findings: Big Data systems are difficult to construct tools and applications thatintegrate data from multiple Big Data sources. The systems should bedesigned with security in mind from the ground up rather than have itemerge as an afterthought. Identified 16 potential standardization gaps to enable Big Data Systemsinteroperability.Organizational Drivers to Provide:1. Insight: enable discovery of deeper, fresher insights from all enterprisedata resources2. Productivity: improve efficiency, effectiveness, and decision-making3. Speed: facilitate more timely, agile response to business opportunities,threats, and challenges4. Breadth: provide a single view of diverse data resources throughout thebusiness chain5. Control: support tighter security, protection, and governance of datathroughout its lifecycle6. Scalability: improve the scale, efficiency, performance, and costeffectiveness of data/analytics platforms* JTC 1 Big Data Report: http://www.iso.org/iso/big data report-jtc1.pdfNIST Big Data Standards Activities, Wo Chang, May 20, 201613

Collaboration Focus AreasJoin Big Data international standards development 140 from 22 NBs: Australia, Austria, Brazil, Canada, China, Finland, France, Germany,India, Ireland, Italy, Japan, Korea, Luxembourg, Netherlands, Norway, RussianFederation, Spain, Singapore, Sweden, UK, US Current Projects ISO/IEC 20546 Information technology – Big data – Overview and vocabulary ISO/IEC 20547 Information Technology – Big data Reference architecture(5 Parts)Part 1: (TR) Framework and Application ProcessPart 2: (TR) Use Cases and Derived RequirementsPart 3: (IS) Reference ArchitecturePart 4: (IS) Security and Privacy Fabric (under SC 27/WG 4)Part 5: (TR) Standards Roadmap ISO/IEC Liaisons: SC 6/WG 7, SC 27, SC 29, SC 32, SC 36, SC 38, SC 39, ISO/TC 69,ISO/TC 204, ITU-T SG13NIST Big Data Standards Activities, Wo Chang, May 20, 201614

Other TopicsMake application domains (Big Data, IoT, CPS, Smart Cities, etc.) available Share public accessible use cases Share public accessible non-PII datasets Share public accessible analytics toolsNIST Big Data Standards Activities, Wo Chang, May 20, 201615