Apache NiFi 0.2.0 发布，数据处理和分发系统

renduoa

腾讯SNG夏日招聘集结号：200个空缺岗位，JAVA、C++、云计算、Android……
　　Apache NiFi 0.2.0 发布了，该项目目前还处于 Apache 基金会的孵化阶段。
　　Apache NiFi 是一个易于使用、功能强大而且可靠的数据处理和分发系统。Apache NiFi 是为数据流设计。它支持高度可配置的指示图的数据路由、转换和系统中介逻辑。
　　架构：


　　集群管理器：

　　主界面：

　　关键特性包括:

• 　　基于web的用户界面
  
  
  
  • 　　无缝体验设计、控制和监视
    
  
  
• 　　高度可配置的
  
  
  
  • 　　数据丢失容错和保证交付
    
  • 　　低延迟和高吞吐量
    
  • 　　动态优先级
    
  • 　　流可以在运行时修改
    
  • 　　背压 Back presure
    
    
  
  
• 　　数据来源
  
  
  
  • 　　从始至终跟踪数据流
    
  
  
• 　　为扩展设计
  
  
  
  • 　　构建自己数据处理器
    
  • 　　支持快速开发和有效的测试
    
  
  
• 　　安全
  
  
  
  • 　　SSL,SSH,HTTPS加密内容,等等……
    
  • 　　可插拔的基于角色的验证/授权
    
  
  

　　主要改进内容包括：
Sub-task

• 　　[NIFI-626] - ExecuteSQL processor for executing arbitrary SQL queries
  

Bug

• 　　[NIFI-16] - Site-to-site should use canonical hostnames when sending peer list
  
• 　　[NIFI-37] - If provenance receive/create/fork/join/clone event registered against flowfile not created in session...
  
• 　　[NIFI-41] - Shoud not be able to change destination of Connection if component has its FlowFiles
  
• 　　[NIFI-80] - When provenance repo retrieves an event that has attributes > some threshold, truncate the value
  
• 　　[NIFI-126]- Mock Framework should detect when session.create() called instead of session.create(FlowFile) / session.create(Collection) without emitting CREATE/RECEIVE Provenance Event
  
• 　　[NIFI-129] - Root Group Ports show that they are not transmitting even when there are active threads receiving data
  
• 　　[NIFI-155] - Test Failure: MulticastProtocolListenerTest and ClusterServiceDiscoveryTest
  
• 　　[NIFI-230] - Volatile Content Repository shows negative values available in 'System Diagnostics'
  
• 　　[NIFI-313] - When template is deleted, UI still shows "Displaying X of Y" instead of "Displaying X of (Y-1)"
  
• 　　[NIFI-334]- When WriteAheadFlowFileRepo is restoring FlowFiles, it needs to look at the swapped flow files, Id's before setting the max id for the FlowFile ID.
  
• 　　[NIFI-373]- If NiFi UI is secure but cluster communication is not, NiFi doesn't start but throws NullPointer instead of nice error message
  
• 　　[NIFI-378] - MergeContent in Defragment mode will merge fragments without checking index
  
• 　　[NIFI-379] - MergeContent does not merge when Maximum number of Bins is set to 1
  
• 　　[NIFI-427] - Too many steps to delete flows
  
• 　　[NIFI-472]- When running NiFi with the "run.as" property specified in the bootstrap.conf file, the run.as user should own the nifi.pid file
  
• 　　[NIFI-482] - users are currently able to evaluate a function against the result of an aggregrate function
  
• 　　[NIFI-487] - JsonPath processors can mark valid JSON as invalid
  
• 　　[NIFI-488] - Bootstrap not capturing output stream or error stream
  
• 　　[NIFI-532] - Exception handling in RunNifi
  
• 　　[NIFI-545] - DataFlowDaoImpl writeDataFlow creates an unused dataflow
  
• 　　[NIFI-546] - StandardDataFlows constructor declears it throws NullPointerException but does not
  
• 　　[NIFI-547] - Data Viewer: Better support for Unicode characters
  
• 　　[NIFI-564] - ReplaceText not working as expected when forming POST requests for InvokeHTTP
  
• 　　[NIFI-565] - System-Level bulletins generated on nodes do not show in the UI when clustered
  
• 　　[NIFI-571] - Findbugs issue in StandardControllerServiceProvider
  
• 　　[NIFI-578] - FlowFileNode does not set clusterNodeIdentifier
  
• 　　[NIFI-580] - Typo in documentation for EncryptContent
  
• 　　[NIFI-582] - Inconsistent naming convention of IDs for the same component in NiFi UIs
  
• 　　[NIFI-591] - CompressContent uses wrong mime type for BZIP2
  
• 　　[NIFI-594] - TestExtractHL7Attributes writes to System output
  
• 　　[NIFI-602] - Error on shutdown bin/nifi.sh stop
  
• 　　[NIFI-613] - ScanContent throws exception on dictionary with blank line
  
• 　　[NIFI-625] - ReplaceText does not substitute Expression Language if embedded within curly braces
  
• 　　[NIFI-630] - Process Group ID
  
• 　　[NIFI-632] - Documentation Generation: Nar ClassLoader
  
• 　　[NIFI-633] - Documentation Generation: Logger
  
• 　　[NIFI-637] - Improve Nifi Windows Batch Scripts
  
• 　　[NIFI-638] - Rest API documentation is not clickable
  
• 　　[NIFI-643] - SplitJson creates a cycle of provenance events
  
• 　　[NIFI-647] - Duplicating provenance reporting via session creates unexpected behavior
  
• 　　[NIFI-657] - MongoDB unit tests fail
  
• 　　[NIFI-661] - Controller Services tab not showing resons for "Invalid" state in Firefox
  
• 　　[NIFI-662] - Unit test failures for ExecuteStreamCommand
  
• 　　[NIFI-663] - TestExecuteStreamCommand tests are not OS safe
  
• 　　[NIFI-664] - WebUI - Invalid argument to setTimeout when disabling ControllerServices
  
• 　　[NIFI-670] - Copying and pasting a processor using a controller service generates copies of the controller service
  
• 　　[NIFI-672] - Documentation: Admin Guide-Controlling Levels of Access example incorrect
  
• 　　[NIFI-674] - FileSystemRepository should not create new threads in its constructor
  
• 　　[NIFI-676] - DBCPService throws NPE if using OtherDB
  
• 　　[NIFI-677] - DBCPService should allow user to specify Connection URL
  
• 　　[NIFI-678] - DBCPService should not require password
  
• 　　[NIFI-682] - EvaluateXPath docs missing closing parenthesis
  
• 　　[NIFI-683] - When connecting to cluster, splash screen remains indefinitely if error sent back
  
• 　　[NIFI-691] - Default config causes NiFi to stop processing after a while if more than 50% of disk used
  
• 　　[NIFI-693] - Cannot download template
  
• 　　[NIFI-694] - If Enabling Controller Service fails, no indication is provided to user
  
• 　　[NIFI-695] - Cancel Enable/Disable controller service should provide immediate feedback to user
  
• 　　[NIFI-699] - Remove Enable checkbox from Controller Service Configuration Dialog
  
• 　　[NIFI-701] - Downloaded templates are missing ControllerServices
  
• 　　[NIFI-702] - PutFile writes too much data when used with VolatileContentRepository
  
• 　　[NIFI-703] - When calling Session.get(int), it's possible to not get any FlowFiles even when queue is not empty
  
• 　　[NIFI-705] - TestHandleHttpRequest#testRequestAddedToService fails in Ubuntu 14.04
  
• 　　[NIFI-711] - Exception thrown when routing FlowFile to multiple connections
  
• 　　[NIFI-717] - nifi-bootstrap.log written to directory relative to current working directory
  
• 　　[NIFI-718] - nifi.sh install does not properly install nifi as a linux service
  
• 　　[NIFI-719] - For Reporting Tasks, ConfigurationContext should  provide scheduling information
  
• 　　[NIFI-720] - If Reporting Task fails to start properly and is then stopped, it can continue to run once it is able to start
  
• 　　[NIFI-721] - Typos in ExtractText
  
• 　　[NIFI-722] - Reporting Task Run Schedule
  
• 　　[NIFI-728] - Unit test failure on multi-core/fast system builds: nifi-distributed-cache-server
  
• 　　[NIFI-731] - If content repo is unable to destroy content as fast as it is generated, nifi performance becomes very erratic
  
• 　　[NIFI-733] - GetKafka group identifier is ignored
  
• 　　[NIFI-734] - GetKafka Kafka Timeout property is ignored
  
• 　　[NIFI-735] - DocGenerator should call @OnShutdown after its done generating the docs
  
• 　　[NIFI-738] - Do not write conversion error messages to flow file content
  
• 　　[NIFI-743] - .getSolr-mock-processor and .httpCache-mock-processor files in conf dir
  
• 　　[NIFI-745] - Disabling Controller Service stuck
  
• 　　[NIFI-749] - Unit tests fail on Windows for InvokeHTTP
  
• 　　[NIFI-753] - NPE when reading in provenance data
  

Improvement

• 　　[NIFI-75] - Mock Framework should fail if session.create() used and no appropriate prov event created...
  
• 　　[NIFI-349] - NiFi should set namespaceAware flag on document builders
  
• 　　[NIFI-413] - KafkaPut should support compression option
  
• 　　[NIFI-419] - SSLContextService should allow configurable TLS version
  
• 　　[NIFI-454] - Distributed cache server unit tests should use random ports
  
• 　　[NIFI-548] - Listen UDP should support generation of a flowfile per datagram
  
• 　　[NIFI-551] - Improve error handling for ConvertJSONToAvro processor
  
• 　　[NIFI-553] - Remove experimental tag from some processors
  
• 　　[NIFI-583] - Provide ExecuteStreamCommand option of streaming contents over STDIN of an incoming flowfile
  
• 　　[NIFI-600] - Add compression support to PutHDFS and GetHDFS
  
• 　　[NIFI-606] - Add a NiFi Storm Spout
  
• 　　[NIFI-618] - Allow components to be taken out of a processor group
  
• 　　[NIFI-642] - Eliminate hardcoded HDFS compression codec classnames.
  
• 　　[NIFI-644] - Data Viewer: Use mime.type when known
  
• 　　[NIFI-645] - Set content archive and content viewer on by default
  
• 　　[NIFI-654] - Update dependency versions
  
• 　　[NIFI-679] - InvokeHTTP - Add support for basic authentication
  
• 　　[NIFI-680] - Processor docs don't always need to mention Sensitive properties or EL
  
• 　　[NIFI-685] - Proxy Support for GetHTTP and PostHTTP processors
  
• 　　[NIFI-704] - StandardProcessorTestRunner should allow you to wait before calling OnUnScheduled methods
  
• 　　[NIFI-707] - GetHTTP should set mime.type attribute based on HTTP Content-Type header
  
• 　　[NIFI-724] - Controller Services and Reporting Tasks should be able to emit bulletins
  
• 　　[NIFI-737] - Typo in developer guide 'lot message'
  
• 　　[NIFI-750] - NiFi Storm Spout should allow FlowFile attributes to be put into Tuple values
  

New Feature

• 　　[NIFI-322] - New Database Connection Pooling Controller Service
  
• 　　[NIFI-570] - MongoDB Processors
  
• 　　[NIFI-601] - Use dynamic properties as environment variables in ExecuteStreamCommand
  

Task

• 　　[NIFI-584] - Remove javadoc author tags
  
• 　　[NIFI-656] - Release Manager: Release 0.2.0
  
• 　　[NIFI-659] - rat-plugin needs to be configured to exclude child and parent items