使用repartitionDS读出的是空表怎么办

准备把date和factor两层分区的数据重新以date分区，逐月提取数据操作，以下代码执行出错， 'port = mr(ds, dataReady::mapFactors{, avg, freq}, , unionAll{, ports}) => dataReady::mapFactors: return dealwork::freqTransform(df1, mean, freq) => dealwork::freqTransform: Factorweek = ::evaluate(sql(colDefs, indata, , [sqlCol("stock"), sqlCol("date", tran_date, "date")], 1)) => Unrecognized column name stock'，找不到stock列，说明pivotBy未执行，或读出的df为空表。不做repartitionDS时用ds=sqlDS(<select * from loadTable("dfs://factors_value","factors_value") where factor in factors , date>=startDate and date <= endDate >) 读取运行正常，但是最终合并表时无法拼接factor列，列不匹配存不进ports库。哪里出了问题？

def mapFactors(df,mean,freq){

return freqTransform(df,mean,freq)

}

def pivotBy(df){

return select value from df pivot by date, stock,factor

}

ds = repartitionDS(<select * from loadTable("dfs://factors_value","factors_value") where factor in factors , date in [2020.01.01,2023.05.01]>, `date,RANGE,2020.01.01 + (0..40)*1)

ds.transDS!(pivotBy)

ports=mr(ds, mapFactors{,mean,freq},,unionAll{,ports})

3 条评论

默认排序时间排序

1 个回答

Polly 2023-10-13 09:31

freqTransform 中列处理逻辑：

    for(i in 0:colCount){ 
        if (factors[i] == `stock){
            colDefs[i] = sqlCol(factors[i])
        } else if (factors[i] == `date){
            colDefs[i] = sqlCol(factors[i],last,`tobedelete)
        } 
        else{ 
            colDefs[i] = sqlCol(factors[i], mode,factors[i]) 
        } 
    }

我初步看了一下，当列是 factor 时走 else 逻辑，最后调用了 mean 计算，返回结果列是 DOUBLE 类型，和原表 SYMBOL 类型不匹配。您明确一下是不是这个问题

使用repartitionDS读出的是空表怎么办

1 个回答

相似问题