qq_37597249-CSDN博客

原创 Spark 2.X 读写 Oracel 代码实例作者：刘大少

import org.apache.spark.sql.{SaveMode, SparkSession}import org.apache.spark.sql.jdbc.JdbcDialectsimport org.apache.spark.sql.jdbc._import org.apache.spark.sql.types._一、读Oracle 表// 参数自己传进来1、定...

2018-09-03 15:29:08 216

原创 spark on yarn 资源计算公式

一、spark on yarn 提交任务有两种模式1、cluster模式 :ApplicationMaster进程进行资源分配和executor的启动提交命令：--master yarn \--deploy-mode client \--driver-memory 5G \ applicationMaster 所在的容器分配的内存--driver-cores 5 \ a...

2020-01-16 14:28:31 738

原创 SPARK-UDF yyyy-MM-dd HH:mm:ss.SSS 类型转秒+微秒的时间戳

session.udf.register("toMMS",(str:String) =>{ val sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS") val sdf2 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss") var st ="" if(str.length>1...

2018-07-16 17:51:18 2545

原创 SPARK报错java.long.Exception……org.codehaus.commons.compiler.CompileException: File 'generated.java'

报错信息： java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 61, Column 30: Index expression of ...

2018-06-22 15:10:00 5856 3

原创 SparkSQL报错:org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:

错误信息：06/13/18 13:17:50 06/14/18 13:17:50 krbtgt/[email protected]/06/13 13:17:59 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under S...

2018-06-13 13:56:20 24030 2

原创 Container killed by YARN for exceeding memory limits. 15.6 GB of 15.5 GB physical memory used. Consi

spark 报错 [Stage 45:(482 + 46) / 528][Stage 46:(64 + 148) / 528][Stage 55:>(27 + 0) / 528]18/06/07 16:07:11 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceedin...

2018-06-07 17:42:45 3843

原创关闭 centos7 的滴滴声

1.执行：vi /etc/inputrc #set bell-style none 把这一行的#去掉2、执行：vi ~/.bashrc 添加：setterm –blength 03、执行：reboot 声音关闭

2018-06-01 11:22:06 2707

原创 spark 给hive表填加自增字段和UUID

直接操作表的时候用自增这个方式会报错因为spark over里必须有字段在hive中使用直接用SQL : create table table2Name as select row_number() over () as row_id,* from table1NameUUID主键：也可以操作表也可以操作RDD 可以同时这种在增加UUID之后必须cache 他会一直到触发a...

2018-06-01 10:32:29 5230

qq_37597249的博客